Answer

AIC vs BIC: which model selection criterion should you use?

Inspired by a question on Cross Validated · September 23, 2026

model selectionregression

Two formulas that differ in one term

Both criteria start from the same fit measure, minus twice the maximized log-likelihood, and add a charge for complexity. With k parameters and n observations:

• AIC = 2k - 2 log L. Each extra parameter costs 2 points.
• BIC = k log(n) - 2 log L. Each extra parameter costs log(n) points, which grows with the sample size.

Lower is better for both. Because log(n) exceeds 2 once n is 8 or more, BIC is the harsher of the two for almost any real dataset, and the gap widens as data accumulate. At n = 1,000 an extra parameter costs about 6.9 points under BIC but still only 2 under AIC. That difference in strictness is not an accident: it follows from the two criteria aiming at different targets.

They are built for different goals

AIC is about prediction. Akaike derived it as an estimate of how far a fitted model is from the process that generated the data (the Kullback-Leibler distance). Choosing the lowest AIC is, in large samples, close to choosing the model with the best leave-one-out cross-validation score. AIC does not assume that any candidate is the true model; it just looks for the most useful approximation.

BIC is about identification. Schwarz derived it as an approximation to the Bayesian evidence for each model, so differences in BIC can be turned into rough posterior model probabilities. Its key property is consistency: if the true model is among your candidates and the number of parameters stays fixed, the probability that BIC picks it goes to 1 as n grows. AIC lacks that property. Even with unlimited data it keeps a steady chance, about 16% per useless term, of adding a variable that does nothing.

The trade-off runs the other way too. When reality is more complex than any model you are considering, which is the usual situation, BIC's heavy penalty can make it too conservative, dropping small but real effects that would have improved predictions.

See the difference in R and Python

This simulation uses a known truth: y depends on x1 and x2, and x3 is pure noise. It compares three nested models on one dataset, then repeats the experiment 1,000 times at three sample sizes to count how often each criterion chooses each model. The R version uses base R only; switch to the Python tab for the same analysis using NumPy.

set.seed(2026)

# One dataset: y depends on x1 and x2; x3 is pure noise
n  <- 100
x1 <- rnorm(n); x2 <- rnorm(n); x3 <- rnorm(n)
y  <- 1 + 0.5 * x1 + 0.3 * x2 + rnorm(n)

fits <- list(m1 = lm(y ~ x1), m2 = lm(y ~ x1 + x2), m3 = lm(y ~ x1 + x2 + x3))
round(cbind(AIC = sapply(fits, AIC), BIC = sapply(fits, BIC)), 1)

# Repeat many times: how often does each criterion pick each model?
pick <- function(n) {
  x1 <- rnorm(n); x2 <- rnorm(n); x3 <- rnorm(n)
  y  <- 1 + 0.5 * x1 + 0.3 * x2 + rnorm(n)
  f  <- list(lm(y ~ x1), lm(y ~ x1 + x2), lm(y ~ x1 + x2 + x3))
  c(AIC = which.min(sapply(f, AIC)), BIC = which.min(sapply(f, BIC)))
}

for (n in c(30, 100, 1000)) {
  res <- replicate(1000, pick(n))
  cat("\nn =", n, "(share of 1000 runs choosing each model)\n")
  print(rbind(
    AIC = table(factor(res["AIC", ], 1:3)) / 1000,
    BIC = table(factor(res["BIC", ], 1:3)) / 1000
  ))
}

import numpy as np

rng = np.random.default_rng(2026)

def aic_bic(y, X):
    # Gaussian linear model; k counts the coefficients plus the residual variance, as R does
    n, p = X.shape
    beta, *_ = np.linalg.lstsq(X, y, rcond=None)
    rss = np.sum((y - X @ beta) ** 2)
    loglik = -n / 2 * (np.log(2 * np.pi) + np.log(rss / n) + 1)
    k = p + 1
    return 2 * k - 2 * loglik, k * np.log(n) - 2 * loglik

def fit_all(n):
    # y depends on x1 and x2; x3 is pure noise
    x1, x2, x3 = rng.normal(size=(3, n))
    y = 1 + 0.5 * x1 + 0.3 * x2 + rng.normal(size=n)
    one = np.ones(n)
    designs = [np.column_stack(c) for c in ([one, x1], [one, x1, x2], [one, x1, x2, x3])]
    return np.array([aic_bic(y, X) for X in designs])  # rows m1-m3, columns AIC, BIC

# One dataset of 100
print(np.round(fit_all(100), 1))

# Repeat many times: how often does each criterion pick each model?
for n in (30, 100, 1000):
    picks = np.array([fit_all(n).argmin(axis=0) for _ in range(1000)])
    print(f"\nn = {n} (share of 1000 runs choosing models 1, 2, 3)")
    for j, name in enumerate(("AIC", "BIC")):
        print(name, np.bincount(picks[:, j], minlength=3) / 1000)

The figures below come from one seeded run of the R code. Python uses a different random number generator, so its figures differ slightly, but the pattern is the same. In the single dataset, AIC is essentially tied between the true model (297.9) and the one with the noise term (297.7), while BIC clearly favors the true model (308.3 versus 310.7). Across repetitions, the share of runs picking the correct model was:

• n = 30: AIC 51%, BIC 40%. BIC dropped the real but modest x2 effect in 55% of runs.
• n = 100: AIC 78%, BIC 76%. AIC added the noise term in 16% of runs, BIC in 3%.
• n = 1,000: AIC 85%, BIC 99%. AIC still added the noise term 15% of the time; BIC almost never did.

That is the whole story in miniature: with little data, BIC's caution costs it real effects; with lots of data, AIC's leniency keeps letting a useless variable in. Note that R's AIC() and BIC() for lm count the residual variance as a parameter too, and the Python function does the same, so the two give identical values on identical data.

Which one to use

• Forecasting or prediction: AIC. A slightly oversized model usually predicts about as well, while a missing effect can hurt.
• Small samples: AICc, which adds 2k(k+1)/(n - k - 1) to AIC. Burnham and Anderson suggest it whenever n/k is below about 40; it converges to AIC as n grows.
• Deciding which predictors truly matter: BIC, especially with large n and a short list of plausible, theory-driven models.
• When they disagree: report both. Disagreement usually means the extra terms are small effects, and the honest conclusion is that the data cannot settle the question firmly.

If prediction is the real goal and you have enough data, direct cross-validation is often better than either criterion, because it measures out-of-sample error without relying on large-sample approximations.

Common mistakes

• Comparing across different data. Models must be fit to the same rows and the same response. Dropping cases with missing values in one model, or modeling log(y) in one and y in another, makes the numbers incomparable.
• Reading the raw value. An AIC of 297.9 means nothing alone. Only differences between models fitted to the same data carry information.
• Mixing software. Programs can drop different constants from the log-likelihood, so compare values only within one program.
• Treating tiny gaps as decisive. A difference under about 2 points is weak evidence either way. For BIC, Kass and Raftery describe gaps of 2-6 as positive, 6-10 as strong, and over 10 as very strong evidence.
• Searching thousands of models. Picking the minimum across a huge automated search overfits, whichever criterion you use.

How to Report (APA 7th)

State which criterion you used and why, give the values for every candidate model, and report differences from the best model (ΔAIC or ΔBIC), usually to one decimal place. Using the single dataset above as an example, an inline write-up could read:

Three nested linear regression models were compared using AIC and BIC. AIC was essentially tied between the two-predictor model (AIC = 297.9) and the model that added x3 (AIC = 297.7, ΔAIC = 0.2). BIC favored the two-predictor model (BIC = 308.3) over the three-predictor model (BIC = 310.7, ΔBIC = 2.4), so the more parsimonious model was retained.

In a table, give one row per model with columns for the predictors included, the number of parameters (k), AIC, ΔAIC, BIC and ΔBIC, and mark the preferred model in a note below the table.

Related tools and guides

• Which statistical test should I use?
• Akaike information criterion (Wikipedia)
• Bayesian information criterion (Wikipedia)

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.