Answer

Is an F-test gate needed before Tukey's HSD?

Inspired by a question on Cross Validated · November 1, 2026

anovamultiple comparisons

The short answer

Many students are taught a two-step ritual: run a one-way ANOVA, and only if the F test is significant, go on to compare the groups. Whether that first step is required depends on which follow-up method you use.

• Tukey's HSD, Dunnett's test, Holm and Bonferroni do not need it. Each was built to hold the familywise error rate (the chance of at least one false positive among the comparisons) at α on its own. Adding an F gate in front makes them stricter than they need to be, so you lose a little power and gain nothing you were promised.
• Fisher's least significant difference (LSD) does need it. LSD compares the pairs with ordinary, unadjusted t tests that share the ANOVA's pooled variance. The significant F is its only protection. That protection is complete with three groups, but with four or more groups it can fail when some means truly differ and others do not.
• **Unadjusted t tests with no gate are the real problem.** With several groups, running every pair at α = .05 makes at least one false positive far more likely than 5%. This is the classic reason people are told to "do an ANOVA instead of lots of t tests".

So the honest version of the rule is: use a procedure that controls the familywise error rate, and then you may skip the F test or report it without letting it decide what you look at. Our page on t tests vs ANOVA with two groups covers the two-group case, where the F test and the t test are the same test; this page is about three or more groups.

See it in R and Python

The code simulates studies with four groups of 15 observations (56 error degrees of freedom) and applies four rules to the six pairs of groups: unadjusted t tests with no gate, Fisher's LSD (unadjusted tests only after a significant F), Tukey's HSD on its own, and Tukey's HSD only after a significant F. It tries three situations, 5,000 studies each: all four population means equal; two pairs of equal means far apart (0, 0, 2, 2, in SD units), where we count false claims about the two equal pairs; and one group 0.8 SD above the other three, where we count how often a rule finds at least one of the real differences.

set.seed(305583)
k <- 4; n <- 15; df_err <- k * n - k        # 4 groups of 15, 56 error df
pairs <- combn(k, 2)                          # the 6 pairs: 12 13 14 23 24 34
crit <- c(F = qf(0.95, k - 1, df_err),        # critical values at alpha = .05
          t = qt(0.975, df_err),
          tukey = qtukey(0.95, k, df_err) / sqrt(2))
round(crit, 3)

# One study: which pairs each rule calls significant, and whether F rejects
one_study <- function(mu) {
  x <- matrix(rnorm(n * k, mean = rep(mu, each = n)), n)
  m <- colMeans(x); mse <- mean(apply(x, 2, var))
  t_abs <- abs(m[pairs[1, ]] - m[pairs[2, ]]) / sqrt(2 * mse / n)
  list(F_sig = n * var(m) / mse > crit[["F"]],
       t_sig = t_abs > crit[["t"]], tukey_sig = t_abs > crit[["tukey"]])
}

# Share of studies in which a rule declares at least one pair in `target` different
rates <- function(mu, target, reps = 5000) {
  out <- replicate(reps, {
    s <- one_study(mu)
    c(t_tests      = any(s$t_sig[target]),              # unadjusted, no F gate
      fisher_lsd   = s$F_sig && any(s$t_sig[target]),   # unadjusted after a significant F
      tukey        = any(s$tukey_sig[target]),          # Tukey on its own
      F_then_tukey = s$F_sig && any(s$tukey_sig[target]))
  })
  rowMeans(out)
}

round(rbind(
  all_equal    = rates(c(0, 0, 0, 0), target = 1:6),    # false positives, full null
  partial_null = rates(c(0, 0, 2, 2), target = c(1, 6)), # false positives, pairs 12 and 34
  power        = rates(c(0, 0, 0, 0.8), target = c(3, 5, 6)) # finds a real difference
), 3)

import numpy as np
from itertools import combinations
from scipy import stats

rng = np.random.default_rng(305583)
k, n = 4, 15
df_err = k * n - k                             # 4 groups of 15, 56 error df
pairs = np.array(list(combinations(range(k), 2)))   # the 6 pairs: 12 13 14 23 24 34
crit = {"F": stats.f.ppf(0.95, k - 1, df_err),  # critical values at alpha = .05
        "t": stats.t.ppf(0.975, df_err),
        "tukey": stats.studentized_range.ppf(0.95, k, df_err) / np.sqrt(2)}
print({key: round(float(v), 3) for key, v in crit.items()})

# Share of studies in which a rule declares at least one pair in `target` different
def rates(mu, target, reps=5000):
    x = rng.normal(np.repeat(mu, n).reshape(k, n), 1, size=(reps, k, n))
    m = x.mean(axis=2); mse = x.var(axis=2, ddof=1).mean(axis=1)
    f_sig = n * m.var(axis=1, ddof=1) / mse > crit["F"]
    t_abs = np.abs(m[:, pairs[:, 0]] - m[:, pairs[:, 1]]) / np.sqrt(2 * mse / n)[:, None]
    t_any = (t_abs[:, target] > crit["t"]).any(axis=1)
    tukey_any = (t_abs[:, target] > crit["tukey"]).any(axis=1)
    return {"t_tests": t_any.mean(),                     # unadjusted, no F gate
            "fisher_lsd": (f_sig & t_any).mean(),        # unadjusted after a significant F
            "tukey": tukey_any.mean(),                   # Tukey on its own
            "F_then_tukey": (f_sig & tukey_any).mean()}

for name, mu, target in [("all_equal", [0, 0, 0, 0], [0, 1, 2, 3, 4, 5]),
                         ("partial_null", [0, 0, 2, 2], [0, 5]),
                         ("power", [0, 0, 0, 0.8], [2, 4, 5])]:
    print(name, {key: round(float(v), 3) for key, v in rates(mu, target).items()})

    F     t tukey 
2.769 2.003 2.648 
             t_tests fisher_lsd tukey F_then_tukey
all_equal      0.191      0.049 0.047        0.039
partial_null   0.100      0.100 0.022        0.022
power          0.806      0.579 0.575        0.553

All figures come from one seeded run of the R code. The critical values involve no randomness, and Python reproduces them exactly. Python's random numbers differ from R's, so its simulated rates differ slightly (for example 0.198, 0.050, 0.050 and 0.042 when all means are equal) but show the same pattern in every row.

What the simulation shows

• **Ungated, unadjusted t tests are the danger.** With all four means equal, at least one of the six comparisons came out significant in 19.1% of studies, nearly four times the nominal 5%. (It is below the 26.5% you would get from six independent tests, because the pairs share groups and are correlated.)
• Tukey already does the job alone. Its false-positive rate was 4.7% with all means equal and 2.2% for the two equal pairs in the second situation. Putting the F test in front pushed the first figure down to 3.9%: stricter than the 5% you asked for, which is the cost of an unnecessary hurdle.
• The gate is all Fisher's LSD has, and with four groups it is not enough. With all means equal, LSD stayed at 4.9%, because the F test rarely rejects. But when two pairs of equal means sat far apart, the F test rejected in practically every study, so the gate did nothing, and the two unadjusted comparisons of equal means produced a false claim in 10.0% of studies, twice the 5% target.
• The gate costs power. When one group really was 0.8 SD higher, Tukey alone found at least one of those differences in 57.5% of studies and gated Tukey in 55.3%. The loss is small here, but it is pure loss. (The 80.6% for ungated t tests is not a fair comparison, since that rule also has a 19.1% false-positive rate.)

When the one-way F test still earns its place

Dropping the gate does not make the overall F test useless. It answers its own question, and sometimes that is exactly the question you have.

• "Do these groups differ at all?" Sometimes you do not care which groups differ: checking whether several production batches, sites or raters give the same average, or whether randomization produced comparable groups. A single F test is the direct answer.
• Spread-out differences. The F statistic pools evidence across all groups, so it can detect a pattern in which several means drift apart a little even when no single pair is convincing on its own. Our page on a significant ANOVA with no significant pairs shows how far apart the two can get.
• Effect size and larger designs. The ANOVA table gives the sums of squares behind eta squared, and in factorial designs the F tests for main effects and interactions are usually the questions of interest rather than a formality.

If your real questions are a few specific comparisons fixed in advance, test those directly as planned contrasts, with a Holm or Bonferroni correction across the planned set if reviewers expect one; you do not need a significant F first. If you want every pair, use Tukey's HSD (Tukey-Kramer for unequal group sizes); for each treatment against one control, use Dunnett's test. Fisher's LSD is defensible only with exactly three groups. For choosing an analysis in the first place, try our test chooser, and the DASS blog has more on ANOVA follow-ups.

How to report Tukey HSD comparisons in APA style (7th edition)

In the method section, name the procedure and say whether the comparisons depended on the omnibus test, so readers know which error rate you controlled:

"All pairwise differences among the four groups were tested with Tukey's HSD, which controls the familywise error rate at .05; these comparisons were not conditional on the omnibus F test."

In the results, give the omnibus test if you ran it, then the adjusted comparisons, using your own figures in place of the placeholders: "Scores differed across conditions, F(3, 56) = x.xx, p = .xxx, η² = .xx. Tukey HSD comparisons showed that group D scored higher than group A, by x.xx points, 95% CI [LL, UL], adjusted p = .xxx." With several pairs, put the differences, adjusted confidence intervals and adjusted p-values in a table and name the adjustment in a table note.

Related tools and guides

• ANOVA power and sample size calculator
• APA 7 formatter for ANOVA
• t test vs ANOVA with two groups
• Planned contrasts vs post hoc tests
• Significant ANOVA but no significant pairwise comparisons
• Bonferroni correction: when should you use it?
• Tukey's range test (Wikipedia)

More answered questions

• How many participants do you need when comparing several treatment arms?
• Significant ANOVA but no significant pairwise comparisons: how can that happen?
• Planned contrasts vs post hoc tests: why do they give different p-values?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.