Answer

Kolmogorov-Smirnov, t test or Mann-Whitney: which detects a change from a baseline?

Inspired by a question on Cross Validated · October 17, 2026

hypothesis testingnonparametrict testdistributions

The short answer

Comparing a new batch of measurements (run times, assay results, scores) with a reference batch, the three usual two-sample tests answer different questions:

• **Welch's t test asks whether the means** differ. It ignores changes in spread or shape that leave the mean alone.
• Mann-Whitney (the Wilcoxon rank-sum test) asks whether a random value from one sample tends to be larger than a random value from the other. A pure increase in spread around the same centre leaves that chance at 50%, so it misses that too.
• Kolmogorov-Smirnov compares the two empirical cumulative distribution functions and uses the largest vertical gap between them, D, as its statistic. With enough data it will detect any difference at all, which is why it is often recommended for "any kind of change".

The catch is the phrase "with enough data". The KS test spreads its sensitivity across every kind of difference, and it is most sensitive near the middle of the distribution, where a change in spread or a change in the tails barely moves the two cumulative curves apart. With small samples it is a weak detector of exactly the changes people hope it will catch. For how the t test and Mann-Whitney compare with each other, see t test or Mann-Whitney in small samples; this page adds the KS test and the question of what you are trying to detect.

See it in R and Python

The code simulates 2,000 comparisons of a baseline sample with a new sample, 20 values each, for four situations: no change, a mean shift of 0.8 SD, a doubling of the SD with the mean unchanged, and a change of shape only (a right-skewed distribution with the same mean and SD). For each it records how often four tests give p < .05: Welch's t, Mann-Whitney, Kolmogorov-Smirnov, and the F test of equal variances. It then analyses one pair of samples in which the true SD has doubled.

set.seed(43461)

# Share of 2,000 simulated studies (n = 20 per group) with p < .05
# x = baseline sample, y = new sample
power3 <- function(gen_y, n = 20, reps = 2000) {
  r <- replicate(reps, {
    x <- rnorm(n); y <- gen_y(n)
    c(welch_t      = t.test(x, y)$p.value < .05,
      mann_whitney = wilcox.test(x, y)$p.value < .05,   # exact, no ties
      ks           = ks.test(x, y)$p.value < .05,       # exact, no ties
      f_variance   = var.test(x, y)$p.value < .05)      # F test of SDs
  })
  round(rowMeans(r), 3)
}

# New samples: same mean 0 and SD 1 unless stated
skew <- function(n) rexp(n) - 1                 # mean 0, SD 1, right-skewed
rbind(
  no_change   = power3(function(n) rnorm(n)),
  mean_shift  = power3(function(n) rnorm(n, mean = 0.8)),
  sd_doubled  = power3(function(n) rnorm(n, sd = 2)),
  shape_only  = power3(skew)
)

# One baseline vs one new sample whose spread has doubled
x <- rnorm(20, mean = 10, sd = 1)
y <- rnorm(20, mean = 10, sd = 2)
round(c(mean_x = mean(x), mean_y = mean(y), sd_x = sd(x), sd_y = sd(y)), 2)
kt <- ks.test(x, y); wt <- t.test(x, y); mw <- wilcox.test(x, y)
vt <- var.test(y, x)                            # ratio of variances, new / baseline
round(c(D = unname(kt$statistic), p_ks = kt$p.value,
        t = unname(wt$statistic), df = unname(wt$parameter), p_t = wt$p.value,
        W = unname(mw$statistic), p_mw = mw$p.value), 3)
round(c(F = unname(vt$statistic), p_F = vt$p.value,
        sd_ratio = sqrt(unname(vt$estimate)), sd_ratio_lo = sqrt(vt$conf.int[1]),
        sd_ratio_hi = sqrt(vt$conf.int[2])), 3)

import numpy as np
from scipy import stats

rng = np.random.default_rng(43461)

def f_test(x, y):
    """Two-sided F test of equal variances; returns (F = var(x)/var(y), p)."""
    F = np.var(x, ddof=1) / np.var(y, ddof=1)
    cdf = stats.f.cdf(F, len(x) - 1, len(y) - 1)
    return F, min(1.0, 2 * min(cdf, 1 - cdf))

# Share of 2,000 simulated studies (n = 20 per group) with p < .05
# x = baseline sample, y = new sample
def power4(gen_y, n=20, reps=2000):
    res = []
    for _ in range(reps):
        x, y = rng.normal(0, 1, n), gen_y(n)
        res.append([stats.ttest_ind(x, y, equal_var=False).pvalue < .05,
                    stats.mannwhitneyu(x, y, method="exact").pvalue < .05,
                    stats.ks_2samp(x, y, method="exact").pvalue < .05,
                    f_test(x, y)[1] < .05])
    return np.round(np.mean(res, axis=0), 3)

# New samples: same mean 0 and SD 1 unless stated
print("columns: welch_t, mann_whitney, ks, f_variance")
print("no_change ", power4(lambda n: rng.normal(0, 1, n)))
print("mean_shift", power4(lambda n: rng.normal(0.8, 1, n)))
print("sd_doubled", power4(lambda n: rng.normal(0, 2, n)))
print("shape_only", power4(lambda n: rng.exponential(1, n) - 1))   # mean 0, SD 1, skewed

# One baseline vs one new sample whose spread has doubled
x, y = rng.normal(10, 1, 20), rng.normal(10, 2, 20)
print("means", round(x.mean(), 2), round(y.mean(), 2),
      "sds", round(x.std(ddof=1), 2), round(y.std(ddof=1), 2))
kt = stats.ks_2samp(x, y, method="exact")
wt = stats.ttest_ind(x, y, equal_var=False)
mw = stats.mannwhitneyu(x, y, method="exact")
print("D", round(kt.statistic, 3), "p_ks", round(kt.pvalue, 3),
      "t", round(wt.statistic, 3), "p_t", round(wt.pvalue, 3),
      "W", mw.statistic, "p_mw", round(mw.pvalue, 3))
F, p_F = f_test(y, x)                                  # new / baseline
lo = F / stats.f.ppf(0.975, 19, 19); hi = F / stats.f.ppf(0.025, 19, 19)
print("F", round(F, 3), "p_F", round(p_F, 3), "sd_ratio", round(np.sqrt(F), 3),
      "CI", round(np.sqrt(lo), 3), round(np.sqrt(hi), 3))

The figures below come from one seeded run of the R code. Python's random numbers differ from R's, so its figures differ slightly but show the same pattern.

• No change. Rejection rates were .054 (Welch), .052 (Mann-Whitney), .039 (KS) and .051 (F). All four hold roughly 5%; the exact KS test is a little conservative because its statistic can take only a few values with 20 per sample.
• Mean shift of 0.8 SD. Power was .685 (Welch), .666 (Mann-Whitney) and .493 (KS). The general-purpose test lost about a quarter of the power of the tests aimed at location. The F test, which looks only at spread, stayed at .048.
• SD doubled, same mean. Welch and Mann-Whitney stayed near 5% (.050 and .053), as expected, since neither looks at spread. The KS test caught the change only 11.0% of the time. The F test caught it 81.6% of the time.
• Shape only (skewed, same mean and SD). Mann-Whitney rejected 7.0% of the time and KS 9.3%, so a pure change of shape is very hard to see with 20 values. The F test rejected 16.7% of the time even though the variances were equal: it assumes normal data and gives false alarms when the new data are skewed.

Python's run showed the same pattern (KS .107 and F .829 when the SD doubled; .503 for KS against .692 for Welch after the mean shift).

In the single pair of samples, the sample SDs were 1.12 (baseline) and 1.74 (new), with means of 9.96 and 9.88. The KS test gave D = 0.200, p = .832; Welch's test gave t(32.42) = 0.16, p = .872; Mann-Whitney gave W = 207, p = .862. The F test gave a variance ratio of 2.42, p = .061, and the ratio of SDs was 1.56 with a 95% CI of [0.98, 2.47]. None of the tests reached .05, but only the interval tells you something useful: the data are compatible with no change and with the true doubling. Python's pair happened to show a larger gap in SDs (0.86 against 2.17), which the F test flagged (p < .001) while the KS test still did not (p = .081).

Comparing against a baseline built from several earlier batches

When the reference is a set of earlier batches, pooling them into one big sample is tempting but often misleading. If each batch has its own mean that drifts a little, the pooled values spread out more than any single batch does. A new batch is then compared with an artificially wide reference, and real changes look ordinary. The pooled values are also not independent draws from one distribution, which every test above assumes.

A better approach treats each batch as one observation of its summary. Compute the mean (and SD) of each of the k earlier batches. If those batch means look roughly normal, a 95% prediction interval for the next batch's mean is the average of the batch means ± t × s × √(1 + 1/k), where s is the SD of the batch means and t is the 97.5th percentile of the t distribution with k − 1 degrees of freedom. A new batch whose mean falls outside that range has moved more than the usual batch-to-batch drift. The same idea applied to the batch SDs flags a change in spread. This is the logic of a control chart: the normal variation between batches is built into the alarm. With only a few earlier batches the interval is wide, honestly reflecting how little is known about the usual drift.

A practical way to set it up

1. List the changes that matter, such as slower on average, more variable, or a long right tail, and give each a size worth acting on.
2. Use a targeted method for each. Welch's t test or a confidence interval for the mean difference for location; a confidence interval for the ratio of SDs, or a robust spread test such as Brown-Forsythe (Levene's test using medians), for variability; a comparison of a high percentile for the tail.
3. Keep KS as a backstop, not the main detector, and do not read a non-significant KS result as evidence that nothing changed. A non-significant result never shows equality on its own; see Is a non-significant result evidence for the null?.
4. Control the false-alarm rate. If you run three tests and flag a batch when any of them rejects, the chance of a false alarm can approach 15% rather than 5%. Decide in advance how you combine them, for example with a Bonferroni split (see Bonferroni correction: when should you use it?).
5. Collect enough data per batch. A power simulation like the one above, run before you fix the batch size, shows what you can realistically detect.

To match other designs to a test, try the test chooser, and for more plain-language guides, see the DASS blog.

How to report a Kolmogorov-Smirnov test in APA style (7th edition)

Give each sample's size and descriptive statistics, the D statistic and its p value, and say whether the p value was exact. Because a non-significant KS test is weak evidence of no change, pair it with an estimate of the change you care about. With the single pair of samples above:

"A two-sample Kolmogorov-Smirnov test found no significant difference between the distributions of the baseline sample (M = 9.96, SD = 1.12, n = 20) and the new sample (M = 9.88, SD = 1.74, n = 20), D = .20, p = .832 (exact). The new sample was more variable, F(19, 19) = 2.42, p = .061, with a ratio of standard deviations of 1.56, 95% CI [0.98, 2.47]."

In the method section, say which changes the comparison was designed to detect, which test was used for each, and how the false-alarm rate was controlled when several tests were run.

Related tools and guides

• Check a reported p-value
• APA 7 formatter for nonparametric tests
• t-test power and sample size calculator
• t test or Mann-Whitney in small samples: how do you choose?
• How do you compare medians when the groups differ in shape and spread?
• Welch vs Student t test: should you just always use Welch?
• Bonferroni correction: when should you use it, and what are the alternatives?
• Is a non-significant result from a large study evidence for the null?

More answered questions

• How do you compare medians when the groups differ in shape and spread?
• Significant ANOVA but no significant pairwise comparisons: how can that happen?
• t tests vs the F test in regression: what is the difference?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.