Answer

Bagging, boosting and stacking: how do these ensemble methods differ?

Inspired by a question on Cross Validated · October 30, 2026

model selectionbootstrapregression

The short answer

An ensemble turns many models into one. The three classic recipes differ in how the members are trained and how their outputs are combined:

• Bagging (bootstrap aggregating): train the same flexible model, usually a deep decision tree, on many bootstrap resamples of the data, independently and in parallel, then average the predictions (or take a majority vote for classes). It mainly reduces variance. A random forest is bagging plus one extra trick.
• Boosting: train simple models, usually shallow trees, one after another, each fitted to what the current ensemble still gets wrong, and add them up with small weights. It mainly reduces bias. AdaBoost and gradient boosting (including XGBoost and LightGBM) are the best-known versions.
• Stacking (stacked generalization): train different kinds of model, then fit a second-level model, the blender or meta-learner, that learns how much to trust each one, using predictions the base models made on data they did not see.

What they share: each builds a better predictor out of imperfect ones, each gives up some interpretability, and each still needs honest validation on held-out data. None of them is best in every problem.

How each one works, and why

Bagging. A fully grown tree has low bias but is unstable: a slightly different sample gives a very different tree. Averaging many such trees keeps the low bias and smooths out the instability. If each of B predictions has variance σ² and any two are correlated by ρ, their average has variance ρσ² + (1 − ρ)σ²/B. More trees shrink the second term towards zero, but the first stays, which is why bagged trees built from the same data level off. Random forests lower ρ by letting each split consider only a random subset of the predictors, so the trees disagree more and their average is steadier. Bagging is hard to overfit by adding trees, and the trees can be fitted in parallel.

Boosting. Start with a crude prediction, such as the mean. Fit a small tree to the residuals (more generally, to the gradient of the loss), add a shrunken copy of it to the prediction, recompute the residuals and repeat. Each step nudges the model towards the patterns it is still missing, so a sum of weak learners can capture structure no single small tree could. Because every step chases the training errors, boosting can overfit if it runs too long; the number of rounds, the learning rate (shrinkage) and the tree depth are tuned, often with early stopping on a validation set.

Stacking. Fit several base models of different types, say a linear regression, a random forest and a boosted model. Split the training data into folds and, for each fold, collect each model's predictions from a fit that excluded that fold. These out-of-fold predictions become the inputs to the blender, often a linear regression, perhaps with weights constrained to be non-negative. Using in-sample predictions instead would teach the blender to trust whichever model overfits most. Stacking pays off when the base models make different kinds of mistakes.

See it in R and Python

The example simulates 500 training and 5,000 test cases with four predictors. The outcome has a strong straight-line effect, a curve and a two-way interaction, plus noise with variance 0.25, which is the lowest test MSE any model could reach. It compares a linear regression, one fully grown tree, 100 bagged trees, 300 boosted depth-2 trees (learning rate 0.1) and a stack that blends the linear model and the bagged trees using 5-fold out-of-fold predictions. Trees come from the rpart package, which ships with R; the Python version uses scikit-learn.

library(rpart)   # tree package that ships with R
set.seed(18891)

sim <- function(n) {   # 4 predictors: one linear effect, a curve and an interaction
  X <- as.data.frame(matrix(runif(n * 4, -1, 1), n, 4))
  X$y <- 3 * X$V1 + sin(3 * X$V2) + 2 * X$V3 * X$V4 + rnorm(n, 0, 0.5)
  X
}
train <- sim(500); test <- sim(5000)
deep  <- rpart.control(cp = 0, minsplit = 2, minbucket = 1, xval = 0)   # fully grown
small <- rpart.control(cp = 0, maxdepth = 2, minsplit = 10, minbucket = 5, xval = 0)
mse <- function(p) mean((test$y - p)^2)

# Bagging: deep trees on bootstrap samples, averaged
bag <- function(d, new, B = 100) rowMeans(sapply(1:B, function(b) {
  t <- rpart(y ~ ., d[sample(nrow(d), replace = TRUE), ], control = deep)
  predict(t, new)}))

# Boosting: small trees fitted one after another to the current residuals
boost <- function(d, new, M = 300, rate = 0.1) {
  f <- rep(mean(d$y), nrow(d)); p <- rep(mean(d$y), nrow(new))
  for (m in 1:M) {
    r <- d; r$y <- d$y - f
    t <- rpart(y ~ ., r, control = small)
    f <- f + rate * predict(t, d); p <- p + rate * predict(t, new)
  }
  p
}

# Stacking: out-of-fold predictions of two base models feed a linear blender
lin <- function(d, new) predict(lm(y ~ ., d), new)
fold <- sample(rep(1:5, length.out = nrow(train)))
oof <- matrix(NA, nrow(train), 2, dimnames = list(NULL, c("lin", "bag")))
for (k in 1:5) {
  tr <- train[fold != k, ]; ho <- train[fold == k, ]
  oof[fold == k, ] <- cbind(lin(tr, ho), bag(tr, ho))
}
w <- coef(lm(train$y ~ oof))
base <- cbind(lin = lin(train, test), bag = bag(train, test))
stack <- w[1] + base %*% w[-1]

single <- predict(rpart(y ~ ., train, control = deep), test)
round(c(linear = mse(base[, "lin"]), single_tree = mse(single),
        bagging = mse(base[, "bag"]), boosting = mse(boost(train, test)),
        stacking = mse(stack)), 3)
round(w, 3)   # blender intercept and weights

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(18891)

def sim(n):   # 4 predictors: one linear effect, a curve and an interaction
    X = rng.uniform(-1, 1, (n, 4))
    y = 3 * X[:, 0] + np.sin(3 * X[:, 1]) + 2 * X[:, 2] * X[:, 3] + rng.normal(0, 0.5, n)
    return X, y

X_tr, y_tr = sim(500)
X_te, y_te = sim(5000)
deep = dict(random_state=0)   # fully grown
small = dict(max_depth=2, min_samples_split=10, min_samples_leaf=5, random_state=0)
mse = lambda p: np.mean((y_te - p)**2)

# Bagging: deep trees on bootstrap samples, averaged
def bag(X, y, new, B=100):
    preds = []
    for b in range(B):
        i = rng.integers(0, len(y), len(y))
        preds.append(DecisionTreeRegressor(**deep).fit(X[i], y[i]).predict(new))
    return np.mean(preds, axis=0)

# Boosting: small trees fitted one after another to the current residuals
def boost(X, y, new, M=300, rate=0.1):
    f = np.full(len(y), y.mean()); p = np.full(len(new), y.mean())
    for m in range(M):
        t = DecisionTreeRegressor(**small).fit(X, y - f)
        f += rate * t.predict(X); p += rate * t.predict(new)
    return p

# Stacking: out-of-fold predictions of two base models feed a linear blender
lin = lambda X, y, new: LinearRegression().fit(X, y).predict(new)
fold = rng.permutation(np.resize(np.arange(5), len(y_tr)))
oof = np.zeros((len(y_tr), 2))
for k in range(5):
    tr, ho = fold != k, fold == k
    oof[ho] = np.column_stack([lin(X_tr[tr], y_tr[tr], X_tr[ho]), bag(X_tr[tr], y_tr[tr], X_tr[ho])])
blender = LinearRegression().fit(oof, y_tr)
base = np.column_stack([lin(X_tr, y_tr, X_te), bag(X_tr, y_tr, X_te)])
stack = blender.predict(base)

single = DecisionTreeRegressor(**deep).fit(X_tr, y_tr).predict(X_te)
print({k: round(v, 3) for k, v in dict(linear=mse(base[:, 0]), single_tree=mse(single),
       bagging=mse(base[:, 1]), boosting=mse(boost(X_tr, y_tr, X_te)), stacking=mse(stack)).items()})
print("blender intercept", round(blender.intercept_, 3), "weights", np.round(blender.coef_, 3))

     linear single_tree     bagging    boosting    stacking 
      0.854       1.256       0.696       0.419       0.685 
(Intercept)      ooflin      oofbag 
      0.003       0.103       0.912

All figures come from one seeded run of the R code. Python's random numbers differ from R's, so its figures differ slightly, but they show the same pattern: the single tree is worst, bagging beats the linear model, stacking edges out bagging, and boosting comes out clearly on top.

What the numbers show

• One deep tree overfits. Its test MSE (1.256) was the worst of all, worse even than the linear model (0.854), which cannot represent the curve or the interaction at all.
• Bagging fixes the instability. Averaging 100 trees grown on bootstrap samples brought the MSE down to 0.696, about 45% lower than the single tree, with no change to the trees themselves.
• Boosting fits the structure best. Three hundred depth-2 trees, each adding a little, reached 0.419, the closest to the noise floor of 0.25. Small trees added together suit an outcome built from simple effects and one two-way interaction, which is exactly what a depth-2 tree can capture.
• Stacking helps a little here. The blender put a weight of 0.912 on the bagged trees and 0.103 on the linear model, and the stack's MSE (0.685) was only slightly below bagging's. Stacking gains most when the base models are similarly good but wrong in different places; here one base model was clearly stronger, so there was little to add.

Which one should you use?

There is no general winner. On tabular data, well-tuned gradient boosting is often the most accurate single method, but it needs tuning and can overfit. Random forests are a strong default that works well with almost no tuning and gives a free error estimate from the cases left out of each bootstrap sample. Stacking is worth the extra work when you already have several good, different models and a small gain in accuracy matters, as in forecasting competitions.

Whatever you choose, compare the candidates on data they were not trained on, using cross-validation or a separate test set, as described in training, validation and test sets explained. The resampling idea behind bagging is the same one explained in why bootstrapping works. For help choosing a classical analysis instead, try the test chooser or browse the DASS blog.

How to report mean squared error in APA style (7th edition)

In the method section, name each ensemble and its main settings (number of trees, tree depth, learning rate, how stacking weights were estimated) and say how test performance was measured. In the results, give the error on held-out data, in the outcome's squared units. Using the example above:

"Five models were compared on a held-out test set of 5,000 cases. Gradient boosting (300 trees of depth 2, learning rate 0.1) had the lowest mean squared error (MSE = 0.419), followed by a stacked ensemble of a linear regression and bagged trees (MSE = 0.685), bagged trees alone (MSE = 0.696), linear regression (MSE = 0.854) and a single decision tree (MSE = 1.256)."

Spell out the abbreviation at first use. With more than a few models, put the error measures in a table with a note naming the evaluation sample, and report the root mean squared error as well if readers will want the error in the outcome's own units.

Related tools and guides

• APA 7 formatter for regression
• Training, validation and test sets explained
• Why does bootstrapping work?
• RMSE vs standard deviation: what is the difference?
• Which statistical test should I use?
• Ensemble learning (Wikipedia)

More answered questions

• Training, validation and test sets: what is each one for?
• AIC vs BIC: which model selection criterion should you use?
• Mean, median or quantile: which point prediction should a model give?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.