Does Stacking Beat Your Best Model?

The theoretical case for stacking

Ensembling is close to gospel in competitive ML: stack a handful of models, feed their predictions into a meta-model that learns how to combine them, and you generally beat any single model in the stack. The theoretical mechanism is specific, not vague: it works when the base models make different kinds of errors. If model A gets right what model B gets wrong, a meta-model can learn to trust each one where it’s reliable. If all the base models make the same errors, there’s nothing complementary to combine, and stacking should buy you little to nothing.

That’s a testable, falsifiable claim, not just “stacking is good,” but specifically “stacking’s benefit scales with how differently your base models err.” So I built two conditions on the same dataset, one designed to give stacking real room to help and one designed to take that room away, and measured what actually happened. Every result below comes from repeated stratified 5-fold cross-validation (3 repeats, 15 total folds) rather than a single train/test split — a robustness fix for a limitation flagged in an earlier benchmark of mine, where single-split numbers can move around more than they should.

Dataset: a synthetic binary classification problem (sklearn.datasets.make_classification, 2,000 samples, 20 features, realistic noise and class overlap) — clearly synthetic, chosen so the base-model diversity could be controlled directly rather than being an accident of whatever real dataset happened to be lying around.

Condition 1 — diverse base models, the setup stacking’s theory is built on: logistic regression (linear), a decision tree (axis-aligned splits), an SVM (kernel-based, a fundamentally different decision boundary), and k-nearest-neighbors (instance-based, no explicit boundary at all). Four genuinely different ways of drawing a decision boundary.

Condition 2 — correlated base models, deliberately working against stacking’s theory: random forest, gradient boosting, and extra-trees — three variants of the same underlying idea (tree ensembles), likely to agree with each other far more than the diverse group does.

diverse_base = [    ("logreg", LogisticRegression(max_iter=2000)),    ("tree", DecisionTreeClassifier(max_depth=8)),    ("svc", SVC(probability=True)),    ("knn", KNeighborsClassifier(n_neighbors=15)),]stacking = StackingClassifier(    estimators=diverse_base,    final_estimator=LogisticRegression(max_iter=2000),    cv=5,)

The results

In the diverse condition, stacking earned its keep. 0.830 versus the best single model’s 0.820 — a real, meaningful gap given the error bars (±0.015 for stacking, ±0.012 for the single model) barely overlap. This is stacking behaving exactly the way its theoretical case predicts: four models making genuinely different kinds of mistakes gave the meta-model real complementary signal to combine.

In the correlated condition, stacking’s edge nearly vanished. 0.835 versus 0.834 — a 0.1-point difference that’s completely swallowed by the standard deviation on either number (±0.014 and ±0.019 respectively). Three tree-ensemble variants agree with each other closely enough that there was almost nothing left for a meta-model to exploit. This is the theory’s own prediction, confirmed directly: no error diversity, no stacking benefit.

The surprise: voting underperformed the best single model in both conditions. Not stacking — plain averaging. 0.809 versus 0.820 in the diverse condition, 0.829 versus 0.834 in the correlated one. Simple voting has no way to down-weight a genuinely weaker model in the mix; it just averages everyone in, including whichever base model is dragging the group down. Stacking’s meta-model, by contrast, learns how much to trust each base model — which is exactly why it didn’t suffer the same fate.

The cost side of the ledger

In the diverse condition, stacking’s 23.7 seconds for a genuine 1-point accuracy gain is a reasonable trade in most real settings. In the correlated condition, it’s a much harder sell: 125.9 seconds — roughly 35x the single model’s training time — for a 0.1-point difference that isn’t statistically distinguishable from zero. That’s not a rounding error in a research paper; in a pipeline that retrains regularly, it’s a real, recurring cost paid for nothing.

Why voting lost even when stacking won

This deserves its own explanation, because it’s the least intuitive result here. Voting and stacking use the same base models — the only difference is how they’re combined. Voting takes a fixed, equal-weighted average of every base model’s prediction. Stacking trains a second model whose entire job is to learn the right weighting, informed by which base models are actually reliable on which kinds of examples.

In the diverse condition, KNN and the plain decision tree were the weakest of the four base models — solid but clearly behind the SVM. Voting has no mechanism to notice that and downweight them; it just averages all four in in equal proportion, and the weaker two drag the ensemble below what the strongest single model achieved on its own. Stacking’s meta-model, trained on out-of-fold predictions from all four, learns something closer to “trust the SVM more, the tree less” — which is precisely the mechanism that let it beat the best single model instead of getting dragged down by the weakest one.

Where this comparison falls short

• One synthetic dataset. The diversity-versus-correlation pattern is a property of ensemble theory in general, not this dataset specifically, but the exact numbers (the size of stacking’s edge, the size of voting’s shortfall) would shift on real, messier data with a different noise profile.

• Base model choices, not an exhaustive search. A different choice of four “diverse” models, or three “correlated” ones, could shift these specific numbers — though the underlying mechanism (diversity creates room for a meta-model to add value) should hold regardless of which specific models embody “diverse” versus “correlated.”

• Binary classification only. Multi-class or regression stacking can behave differently, particularly around how the meta-model’s cross-validation is structured.

The actual decision framework

Before reaching for a stacking ensemble over your single best model, three questions:

1. Do your candidate base models actually make different kinds of errors, or do they just have different names? Three tree-ensemble variants sound diverse but often aren’t, in the way that actually matters for stacking. Check correlation between base models’ predictions (or errors) before assuming diversity — it’s one line of code and it directly predicts how much stacking will help.

2. Is the accuracy gain, if any, larger than your fold-to-fold noise? A single train/test split will happily report a stacking “win” that’s actually just noise — cross-validate and look at the spread, not just the mean, before trusting a comparison like this one.

3. What’s the actual cost of retraining a stack versus a single model, at your real retraining frequency? 126 seconds once is nothing. 126 seconds every night for a year, for a gain within the noise floor, is a cost worth naming explicitly rather than paying by default.

The takeaway

“Stacking beats your best single model” is true under a specific, checkable condition — genuine diversity in how the base models err — and considerably less true outside it. The more interesting and more actionable finding here isn’t stacking versus single models in the abstract; it’s that naive voting can actually underperform your best individual model even when stacking, using the exact same base models, doesn’t. Combining models isn’t free, and averaging isn’t the same as combining well — the meta-model’s ability to learn who to trust and when is doing real work that a simple average can’t replicate.

Jupyter Notebook for the experiment