AskStats

Answer

Significant ANOVA but no significant pairwise comparisons: how can that happen?

Inspired by a question on Cross Validated ·

anovamultiple comparisonshypothesis testingt test

The short answer

The F test and the pairwise tests ask different questions. With equal group sizes, F is the average of the squared pairwise t statistics, so it pools weak evidence from every pair, while each pair must stand alone and, after a Tukey or Bonferroni adjustment, clear a higher bar. When F has p = .05, the strongest pair's Tukey-adjusted p can be as high as .18 with 4 groups and .68 with 10.

The short answer

It is not a contradiction. The one-way ANOVA F test asks one broad question: are the group means all equal? A set of pairwise comparisons asks many narrow ones: does A differ from B, A from C, and so on. Evidence that is too thin to convict any single pair can still add up to strong evidence that the means are not all the same, and the F test is built to add it up.

Two things widen the gap. First, each pairwise test uses only part of the information. Second, follow-up tests are usually adjusted for multiplicity (Tukey's HSD, Bonferroni), which raises the bar each pair must clear. The opposite surprise happens too: one pair can be significant while the overall F is not, typically when a single group stands apart from many similar ones and the F test spreads that one difference over all the groups.

Why the omnibus test and the pairs can disagree

There is a neat identity behind this. When every group has the same size and each pairwise t uses the ANOVA's pooled error variance (the mean square error), **the F statistic equals the average of the squared pairwise t statistics**. With 4 groups there are 6 pairs, and F is the mean of their 6 values of t². So F is a summary of all the pairwise evidence at once.

Because an average can never exceed its largest member, the strongest pair always has t² at least as big as F. But the two are judged against different yardsticks. F is compared with an F distribution on k − 1 numerator degrees of freedom, whose critical value falls as the number of groups k grows, while a single pair is compared with a t distribution (or, under Tukey's method, with the studentized range for k groups, whose critical value rises with k). That is how F can cross its line while no pair crosses its own.

A more general way to see it: the F test rejects exactly when at least one contrast is significant by Scheffé's method, where a contrast is any weighted comparison of means, such as the average of groups A and B against the average of C and D. The contrast that drives a significant F is often not a simple pair, so a pairwise follow-up can miss it. The same logic explains why a multiple regression can have a significant overall F while no single coefficient is significant.

How far apart can the p values get?

For equal group sizes and the pooled error term, there is an exact worst case. Given a fixed F, the strongest pair is as weak as it can be when the group means split into two clusters of (nearly) equal size, half at one value and half at another. Then the largest squared t is F × k(k − 1)/(2ab), where a and b are the two cluster sizes. The R code below works this out for an F sitting exactly at p = .05, with 10 per group:

So with the pooled error term and unadjusted comparisons (Fisher's LSD), the gap is small: for 3 to 5 groups of this size, a significant F guarantees at least one unadjusted pair below .05. After a multiplicity adjustment the gap can be large, and it grows with the number of groups. Ordinary two-sample t tests that estimate the variance from only the two groups involved loosen the bound further, since each has fewer degrees of freedom. With unequal group sizes the exact figures change.

See it in R and Python

The code checks the identity, computes the worst-case table, confirms the bound by brute force, counts how often the F test and Tukey's HSD disagree in two simulated designs (4,000 data sets each, all groups normal with SD = 1 and 8 per group), and then finds a data set where F is significant but no Tukey pair is.

R

set.seed(83131)

# 1. With equal n and the pooled error term, F is the average squared pairwise t
k <- 4; n <- 8; df <- k * n - k
g <- factor(rep(LETTERS[1:k], each = n))
y <- rnorm(k * n, rep(c(0, 0.4, 0.8, 1.2), each = n))
fit <- aov(y ~ g)
m <- tapply(y, g, mean); mse <- sum(residuals(fit)^2) / df
t2 <- (outer(m, m, "-")[lower.tri(diag(k))])^2 / (2 * mse / n)
round(c(F = summary(fit)[[1]]$`F value`[1], mean_t_squared = mean(t2)), 4)

# 2. Worst case: F sits exactly at p = .05. How weak can the strongest pair be?
worst <- function(k, n = 10) {
  df <- k * n - k; Fc <- qf(0.95, k - 1, df)
  a <- floor(k / 2); b <- k - a           # means split between two values
  t <- sqrt(Fc * k * (k - 1) / (2 * a * b))
  c(k = k, df = df, F_crit = Fc, max_t = t,
    p_unadjusted = 2 * pt(t, df, lower.tail = FALSE),
    p_tukey = ptukey(t * sqrt(2), k, df, lower.tail = FALSE))
}
round(t(sapply(c(3, 4, 5, 6, 8, 10), worst)), 3)

# Check the bound: largest t^2 / mean t^2 over 20,000 random sets of 4 means
ratio <- replicate(20000, { d <- dist(rnorm(4))^2; max(d) / mean(d) })
round(min(ratio), 3)

# 3. How often do the omnibus F and Tukey's HSD disagree? (4,000 data sets)
sim <- function(mu, n = 8, reps = 4000) {
  k <- length(mu); df <- k * n - k
  r <- replicate(reps, {
    y <- matrix(rnorm(k * n, rep(mu, each = n)), n)   # one column per group
    m <- colMeans(y); mse <- sum(sweep(y, 2, m)^2) / df
    pF <- pf(n * sum((m - mean(m))^2) / (k - 1) / mse, k - 1, df, lower.tail = FALSE)
    pT <- ptukey(diff(range(m)) / sqrt(mse / n), k, df, lower.tail = FALSE)
    c(F_only = pF < .05 & pT >= .05, tukey_only = pF >= .05 & pT < .05,
      both = pF < .05 & pT < .05)
  })
  round(rowMeans(r), 3)
}
sim(c(0, 0.4, 0.8, 1.2))         # four means spread evenly
sim(c(0, 0, 0, 0, 0, 1.2))       # one group out of six stands apart

# 4. One data set where F is significant but no Tukey pair is
repeat {
  y <- rnorm(k * n, rep(c(0, 0.4, 0.8, 1.2), each = n)); fit <- aov(y ~ g)
  pF <- summary(fit)[[1]]$`Pr(>F)`[1]; tk <- TukeyHSD(fit)$g
  if (pF < .05 && all(tk[, "p adj"] >= .05)) break
}
summary(fit)
ss <- summary(fit)[[1]]$`Sum Sq`; round(c(eta_squared = ss[1] / sum(ss)), 3)
round(tk, 4)
round(cbind(M = tapply(y, g, mean), SD = tapply(y, g, sd)), 2)

Python

import numpy as np
from scipy import stats

rng = np.random.default_rng(83131)
tk_sf = stats.studentized_range.sf     # Tukey's studentized range

# 1. With equal n and the pooled error term, F is the average squared pairwise t
k, n = 4, 8
df = k * n - k
mu = np.array([0, 0.4, 0.8, 1.2])
y = rng.normal(np.repeat(mu, n), 1).reshape(k, n)    # one row per group
F = stats.f_oneway(*y).statistic
m = y.mean(axis=1); mse = ((y - m[:, None])**2).sum() / df
i, j = np.triu_indices(k, 1)
t2 = (m[i] - m[j])**2 / (2 * mse / n)
print("F", round(F, 4), "mean t^2", round(t2.mean(), 4))

# 2. Worst case: F sits exactly at p = .05. How weak can the strongest pair be?
for kk in [3, 4, 5, 6, 8, 10]:
    d = kk * 10 - kk; Fc = stats.f.isf(0.05, kk - 1, d)
    a = kk // 2; b = kk - a                     # means split between two values
    t = np.sqrt(Fc * kk * (kk - 1) / (2 * a * b))
    print(kk, d, round(Fc, 3), round(t, 3), round(2 * stats.t.sf(t, d), 3),
          round(tk_sf(t * np.sqrt(2), kk, d), 3))

# Check the bound: largest t^2 / mean t^2 over 20,000 random sets of 4 means
z = rng.normal(size=(20000, 4))
d2 = (z[:, i] - z[:, j])**2
print("min ratio", round((d2.max(axis=1) / d2.mean(axis=1)).min(), 3))

# 3. How often do the omnibus F and Tukey's HSD disagree? (4,000 data sets)
def sim(mu, n=8, reps=4000):
    mu = np.asarray(mu); k = len(mu); df = k * n - k
    y = rng.normal(mu[None, :, None], 1, size=(reps, k, n))
    m = y.mean(axis=2); mse = ((y - m[..., None])**2).sum(axis=(1, 2)) / df
    sF = stats.f.sf(n * m.var(axis=1, ddof=1) / mse, k - 1, df) < .05
    # Tukey p < .05 exactly when the range statistic beats its critical value
    sT = np.ptp(m, axis=1) / np.sqrt(mse / n) > stats.studentized_range.isf(.05, k, df)
    return {"F_only": round(np.mean(sF & ~sT), 3),
            "tukey_only": round(np.mean(~sF & sT), 3),
            "both": round(np.mean(sF & sT), 3)}

print(sim([0, 0.4, 0.8, 1.2]))         # four means spread evenly
print(sim([0, 0, 0, 0, 0, 1.2]))       # one group out of six stands apart

# 4. One data set where F is significant but no Tukey pair is
q_crit = stats.studentized_range.isf(.05, k, df)
while True:
    y = rng.normal(np.repeat(mu, n), 1).reshape(k, n)
    a = stats.f_oneway(*y); m = y.mean(axis=1)
    q = np.ptp(m) / np.sqrt(((y - m[:, None])**2).sum() / df / n)
    if a.pvalue < .05 and q < q_crit:
        break
tk = stats.tukey_hsd(*y)
ss_b = n * ((m - m.mean())**2).sum(); ss_t = ((y - y.mean())**2).sum()
print("F", round(a.statistic, 3), "p", round(a.pvalue, 4), "eta^2", round(ss_b / ss_t, 3))
print("Tukey p", np.round(tk.pvalue[i, j], 4))
print("means", np.round(y.mean(axis=1), 2), "sds", np.round(y.std(axis=1, ddof=1), 2))

The figures below come from one seeded run of the R code. The worst-case table involves no randomness, so Python reproduces it exactly. Python's random numbers differ from R's, so its simulated rates and its example data set differ slightly, but they show the same pattern.

What to do when they disagree

Start from the question you actually care about. If your hypotheses are about specific pairs, go straight to a pairwise procedure: Tukey's HSD controls the familywise error rate on its own and does not need a significant F first. If you have a few comparisons planned in advance, test them as planned contrasts, which are more powerful than all-pairs tests; see Planned contrasts vs post hoc tests. If your question really is whether the groups differ at all, the F test answers it, and a non-significant follow-up does not undo that.

Either way, report both results honestly, and lean on estimates rather than the .05 line. A p of .0504 and a p of .0496 are practically the same evidence; the mean differences and their confidence intervals show what the data can and cannot pin down. With only two groups none of this arises, because the F test and the t test are then the same test (see t test vs ANOVA with two groups); the gap only opens up with three or more groups. For why unadjusted pairwise tests inflate false positives, see Bonferroni correction: when to use it. To pick a test for your design, try the test chooser, and the DASS blog has more plain-language guides.

How to report a one-way ANOVA and Tukey HSD in APA style (7th edition)

Report the omnibus result and the follow-up tests as they came out, without calling one of them wrong. With the example above:

"A one-way ANOVA showed that scores differed across the four groups, F(3, 28) = 3.77, p = .022, η² = .29. Tukey's HSD comparisons did not identify any single pair that differed reliably; the largest difference, between groups D (M = 1.44, SD = 1.13) and A (M = −0.07, SD = 1.30), was 1.51, 95% CI [−0.00, 3.03], p = .050."

Give every group's mean, standard deviation and n, in the text or a table. Name the follow-up procedure in the method section ("Pairwise differences were examined with Tukey's HSD"), and drop the leading zero on p and η². When a rounded p lands on .050, as here, make clear it did not fall below .05.

Related tools and guides

More answered questions

Working with your own data?

General answers only go so far. Send us your situation and we'll reply by email within two business days.

Ask your question

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.