Answer

Which model do you keep after k-fold cross-validation?

Inspired by a question on Cross Validated ·

model selectionregression

The short answer

None of the K fold models. Cross-validation is a way to estimate how well a modelling procedure (a model form plus its settings) predicts new data. The K fitted models are temporary by-products of that estimate. Use the cross-validated error to choose between procedures, then fit the chosen procedure once on the full data set; that refit is the model you present, and the cross-validated error is a slightly pessimistic estimate of its performance.

The short answer

It is natural to think that k-fold cross-validation produces K candidate models and that your job is to pick the best one. That is not what it is for. Cross-validation evaluates a procedure: "fit a quadratic regression", "fit a random forest with 500 trees", "fit a lasso with the penalty chosen like this". Each fold fits that same procedure to a slightly smaller training set and scores it on the part it did not see. Averaging over the folds gives an estimate of how well the procedure predicts data it was not trained on.

Once you have that estimate, the K fold models have done their job and can be thrown away. The usual workflow is:

  1. Run cross-validation for every candidate procedure you are considering, using the same folds for all of them.
  2. Choose the procedure with the best cross-validated error (or the simplest one that is nearly as good).
  3. Fit the chosen procedure once on all the data. That refit is your final model.
  4. Report the cross-validated error as the estimate of how that final model will perform.

Why not pick the best fold, or the median fold?

The K fold models are all fits of the same procedure to heavily overlapping data. With 10 folds, any two training sets share about 89% of their rows (8 of the 9 tenths each one uses). Their differences come mostly from which few cases happened to be left out, so they are noisy versions of the same thing rather than genuinely different models.

Picking the fold with the lowest held-out error mostly picks the fold whose test cases were easiest to predict. That says something about those cases, not about the model, and the low error will not carry over to new data. Picking the median fold avoids the most extreme luck but still throws away a tenth of your data for no benefit. Each fold model was trained on only (K − 1)/K of the sample, so the refit on all n cases is usually at least as good, and slightly better when data are scarce.

This is also why the cross-validated error is a little pessimistic for the final model: it was measured on models trained with fewer cases. With 5 or 10 folds the difference is usually small.

See it in R and Python

The code simulates 100 cases from a curved (quadratic) relationship with noise standard deviation 1, so no model can achieve a root-mean-square error (RMSE) much below 1 on new data. It uses 10-fold cross-validation to compare polynomials of degree 1 to 4, then shows how much the 10 fold models of the winning degree differ, and finally refits that degree on all 100 cases.

R

set.seed(52274)

# Simulated data: a curved (quadratic) trend plus noise with SD 1
n <- 100
x <- runif(n, 0, 4)
d <- data.frame(x = x, y = 2 + 1.5 * x - 0.4 * x^2 + rnorm(n, sd = 1))

K <- 10
folds <- sample(rep(1:K, length.out = n))   # assign each row to one fold

# 10-fold CV for a polynomial of a given degree:
# returns the CV RMSE and the coefficients of each of the K fold models
cv_poly <- function(deg) {
  pred <- numeric(n)
  coefs <- matrix(NA, K, deg + 1)
  for (k in 1:K) {
    train <- d[folds != k, ]
    fit <- lm(y ~ poly(x, deg, raw = TRUE), data = train)
    pred[folds == k] <- predict(fit, newdata = d[folds == k, ])
    coefs[k, ] <- coef(fit)
  }
  list(rmse = sqrt(mean((d$y - pred)^2)), coefs = coefs)
}

# Step 1: use CV to compare candidate procedures (degrees 1 to 4)
cv <- lapply(1:4, cv_poly)
cv_rmse <- sapply(cv, `[[`, "rmse")
names(cv_rmse) <- paste0("degree", 1:4)
round(cv_rmse, 3)
best <- which.min(cv_rmse)
best

# The K fold models for the winning degree differ only a little
round(apply(cv[[best]]$coefs, 2, range), 3)   # min and max of each coefficient

# Step 2: refit the winning degree on ALL the data; this is the final model
final <- lm(y ~ poly(x, best, raw = TRUE), data = d)
round(coef(final), 3)

Python

import numpy as np

rng = np.random.default_rng(52274)

# Simulated data: a curved (quadratic) trend plus noise with SD 1
n = 100
x = rng.uniform(0, 4, n)
y = 2 + 1.5 * x - 0.4 * x**2 + rng.normal(0, 1, n)

K = 10
folds = rng.permutation(np.resize(np.arange(1, K + 1), n))  # assign each row to one fold

# 10-fold CV for a polynomial of a given degree:
# returns the CV RMSE and the coefficients of each of the K fold models
def cv_poly(deg):
    pred = np.empty(n)
    coefs = np.empty((K, deg + 1))
    for k in range(1, K + 1):
        tr, te = folds != k, folds == k
        b = np.polynomial.polynomial.polyfit(x[tr], y[tr], deg)  # intercept first, like R
        pred[te] = np.polynomial.polynomial.polyval(x[te], b)
        coefs[k - 1] = b
    return {"rmse": float(np.sqrt(np.mean((y - pred) ** 2))), "coefs": coefs}

# Step 1: use CV to compare candidate procedures (degrees 1 to 4)
cv = [cv_poly(deg) for deg in range(1, 5)]
cv_rmse = {f"degree{d}": round(c["rmse"], 3) for d, c in zip(range(1, 5), cv)}
print(cv_rmse)
best = int(np.argmin([c["rmse"] for c in cv])) + 1
print("best degree:", best)

# The K fold models for the winning degree differ only a little
c = cv[best - 1]["coefs"]
print(np.round(np.vstack([c.min(axis=0), c.max(axis=0)]), 3))  # min and max of each coefficient

# Step 2: refit the winning degree on ALL the data; this is the final model
final = np.polynomial.polynomial.polyfit(x, y, best)
print(np.round(final, 3))

Output from one seeded run of the R code:

degree1 degree2 degree3 degree4 
  1.096   1.007   1.016   1.015 
degree2 
      2 
      [,1]  [,2]   [,3]
[1,] 1.743 1.160 -0.428
[2,] 2.218 1.579 -0.332
               (Intercept) poly(x, best, raw = TRUE)1 
                     1.916                      1.456 
poly(x, best, raw = TRUE)2 
                    -0.389

What the numbers show

Practical points

For how cross-validation relates to a fixed validation and test split, see the page on training, validation and test sets; this page is about what to do with the folds once cross-validation has run.

How to report cross-validated RMSE in APA style (7th edition)

Describe the procedure in the method section (number of folds, candidate models, error measure) and give the cross-validated errors in the results, with the final model's estimates from the full-sample refit. Using the R example above:

"Polynomial regressions of degree 1 to 4 were compared using 10-fold cross-validation, with root-mean-square error (RMSE) as the criterion. The quadratic model had the lowest cross-validated error (RMSE = 1.01; linear model: RMSE = 1.10) and was refitted to the full sample (N = 100), giving a linear coefficient of b = 1.46 and a quadratic coefficient of b = -0.39."

When several models are compared, a small table listing each candidate with its cross-validated RMSE is clearer than running text. State that the fold assignment was random and give the seed or the number of repetitions so others can reproduce it.

Related tools and guides

More answered questions

Working with your own data?

General answers only go so far. Send us your situation and we'll reply by email within two business days.

Ask your question

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.