Answer
Which model do you keep after k-fold cross-validation?
The short answer
None of the K fold models. Cross-validation is a way to estimate how well a modelling procedure (a model form plus its settings) predicts new data. The K fitted models are temporary by-products of that estimate. Use the cross-validated error to choose between procedures, then fit the chosen procedure once on the full data set; that refit is the model you present, and the cross-validated error is a slightly pessimistic estimate of its performance.
The short answer
It is natural to think that k-fold cross-validation produces K candidate models and that your job is to pick the best one. That is not what it is for. Cross-validation evaluates a procedure: "fit a quadratic regression", "fit a random forest with 500 trees", "fit a lasso with the penalty chosen like this". Each fold fits that same procedure to a slightly smaller training set and scores it on the part it did not see. Averaging over the folds gives an estimate of how well the procedure predicts data it was not trained on.
Once you have that estimate, the K fold models have done their job and can be thrown away. The usual workflow is:
- Run cross-validation for every candidate procedure you are considering, using the same folds for all of them.
- Choose the procedure with the best cross-validated error (or the simplest one that is nearly as good).
- Fit the chosen procedure once on all the data. That refit is your final model.
- Report the cross-validated error as the estimate of how that final model will perform.
Why not pick the best fold, or the median fold?
The K fold models are all fits of the same procedure to heavily overlapping data. With 10 folds, any two training sets share about 89% of their rows (8 of the 9 tenths each one uses). Their differences come mostly from which few cases happened to be left out, so they are noisy versions of the same thing rather than genuinely different models.
Picking the fold with the lowest held-out error mostly picks the fold whose test cases were easiest to predict. That says something about those cases, not about the model, and the low error will not carry over to new data. Picking the median fold avoids the most extreme luck but still throws away a tenth of your data for no benefit. Each fold model was trained on only (K − 1)/K of the sample, so the refit on all n cases is usually at least as good, and slightly better when data are scarce.
This is also why the cross-validated error is a little pessimistic for the final model: it was measured on models trained with fewer cases. With 5 or 10 folds the difference is usually small.
See it in R and Python
The code simulates 100 cases from a curved (quadratic) relationship with noise standard deviation 1, so no model can achieve a root-mean-square error (RMSE) much below 1 on new data. It uses 10-fold cross-validation to compare polynomials of degree 1 to 4, then shows how much the 10 fold models of the winning degree differ, and finally refits that degree on all 100 cases.
R
set.seed(52274)
# Simulated data: a curved (quadratic) trend plus noise with SD 1
n <- 100
x <- runif(n, 0, 4)
d <- data.frame(x = x, y = 2 + 1.5 * x - 0.4 * x^2 + rnorm(n, sd = 1))
K <- 10
folds <- sample(rep(1:K, length.out = n)) # assign each row to one fold
# 10-fold CV for a polynomial of a given degree:
# returns the CV RMSE and the coefficients of each of the K fold models
cv_poly <- function(deg) {
pred <- numeric(n)
coefs <- matrix(NA, K, deg + 1)
for (k in 1:K) {
train <- d[folds != k, ]
fit <- lm(y ~ poly(x, deg, raw = TRUE), data = train)
pred[folds == k] <- predict(fit, newdata = d[folds == k, ])
coefs[k, ] <- coef(fit)
}
list(rmse = sqrt(mean((d$y - pred)^2)), coefs = coefs)
}
# Step 1: use CV to compare candidate procedures (degrees 1 to 4)
cv <- lapply(1:4, cv_poly)
cv_rmse <- sapply(cv, `[[`, "rmse")
names(cv_rmse) <- paste0("degree", 1:4)
round(cv_rmse, 3)
best <- which.min(cv_rmse)
best
# The K fold models for the winning degree differ only a little
round(apply(cv[[best]]$coefs, 2, range), 3) # min and max of each coefficient
# Step 2: refit the winning degree on ALL the data; this is the final model
final <- lm(y ~ poly(x, best, raw = TRUE), data = d)
round(coef(final), 3)Python
import numpy as np
rng = np.random.default_rng(52274)
# Simulated data: a curved (quadratic) trend plus noise with SD 1
n = 100
x = rng.uniform(0, 4, n)
y = 2 + 1.5 * x - 0.4 * x**2 + rng.normal(0, 1, n)
K = 10
folds = rng.permutation(np.resize(np.arange(1, K + 1), n)) # assign each row to one fold
# 10-fold CV for a polynomial of a given degree:
# returns the CV RMSE and the coefficients of each of the K fold models
def cv_poly(deg):
pred = np.empty(n)
coefs = np.empty((K, deg + 1))
for k in range(1, K + 1):
tr, te = folds != k, folds == k
b = np.polynomial.polynomial.polyfit(x[tr], y[tr], deg) # intercept first, like R
pred[te] = np.polynomial.polynomial.polyval(x[te], b)
coefs[k - 1] = b
return {"rmse": float(np.sqrt(np.mean((y - pred) ** 2))), "coefs": coefs}
# Step 1: use CV to compare candidate procedures (degrees 1 to 4)
cv = [cv_poly(deg) for deg in range(1, 5)]
cv_rmse = {f"degree{d}": round(c["rmse"], 3) for d, c in zip(range(1, 5), cv)}
print(cv_rmse)
best = int(np.argmin([c["rmse"] for c in cv])) + 1
print("best degree:", best)
# The K fold models for the winning degree differ only a little
c = cv[best - 1]["coefs"]
print(np.round(np.vstack([c.min(axis=0), c.max(axis=0)]), 3)) # min and max of each coefficient
# Step 2: refit the winning degree on ALL the data; this is the final model
final = np.polynomial.polynomial.polyfit(x, y, best)
print(np.round(final, 3))Output from one seeded run of the R code:
degree1 degree2 degree3 degree4
1.096 1.007 1.016 1.015
degree2
2
[,1] [,2] [,3]
[1,] 1.743 1.160 -0.428
[2,] 2.218 1.579 -0.332
(Intercept) poly(x, best, raw = TRUE)1
1.916 1.456
poly(x, best, raw = TRUE)2
-0.389
What the numbers show
- Cross-validation chose the right form. The quadratic had the lowest cross-validated RMSE (1.007), close to the noise level of 1. The straight line was clearly worse (1.096), and degrees 3 and 4 were slightly worse (1.016 and 1.015) because their extra terms only fit noise.
- The fold models are near-copies. Across the 10 quadratic fold models the intercept ranged from 1.743 to 2.218, the linear term from 1.160 to 1.579, and the squared term from -0.428 to -0.332. None of them is special; they scatter around the same curve.
- The final model uses every case. Refitting the quadratic on all 100 rows gave coefficients 1.916, 1.456 and -0.389, close to the true values of 2, 1.5 and -0.4.
- Python agrees on the pattern. Its random numbers differ from R's, so its figures differ slightly (degree 2 again wins, with RMSE 0.863), but it shows the same choice and the same tight spread of fold models.
Practical points
- Do every data-driven step inside the folds. Variable selection, imputation, scaling and tuning must be repeated within each training fold. If you select predictors on the full data first and cross-validate afterwards, the estimate is too optimistic.
- The winner's score is optimistic too. Choosing the minimum of several cross-validated errors favours a procedure that was a bit lucky. When you compare many candidates, report performance from a separate test set or use nested cross-validation, where an outer loop scores the whole choose-then-fit process.
- Prefer simplicity when errors are close. Differences like 1.007 versus 1.015 are within noise. A common convention is the one-standard-error rule: pick the simplest procedure whose error is within one standard error of the best.
- Repeat with new folds if results wobble. The fold assignment is random. Repeating 10-fold cross-validation several times with different splits and averaging makes the comparison more stable.
For how cross-validation relates to a fixed validation and test split, see the page on training, validation and test sets; this page is about what to do with the folds once cross-validation has run.
How to report cross-validated RMSE in APA style (7th edition)
Describe the procedure in the method section (number of folds, candidate models, error measure) and give the cross-validated errors in the results, with the final model's estimates from the full-sample refit. Using the R example above:
"Polynomial regressions of degree 1 to 4 were compared using 10-fold cross-validation, with root-mean-square error (RMSE) as the criterion. The quadratic model had the lowest cross-validated error (RMSE = 1.01; linear model: RMSE = 1.10) and was refitted to the full sample (N = 100), giving a linear coefficient of b = 1.46 and a quadratic coefficient of b = -0.39."
When several models are compared, a small table listing each candidate with its cross-validated RMSE is clearer than running text. State that the fold assignment was random and give the seed or the number of repetitions so others can reproduce it.
Related tools and guides
- APA 7 formatter for regression
- Training, validation and test sets: what is each one for?
- AIC vs BIC: which should you use?
- Which statistical test should I use?
- Cross-validation (Wikipedia)
More answered questions
- When can you drop the intercept from a linear regression?
- Which interaction terms belong in a regression or mixed model?
- Bagging, boosting and stacking: how do these ensemble methods differ?
Working with your own data?
General answers only go so far. Send us your situation and we'll reply by email within two business days.
Ask your questionWritten by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.