Answer

What counts as a family when you correct for multiple comparisons?

Inspired by a question on Cross Validated · November 6, 2026

multiple comparisonshypothesis testingp-values

The short answer

A familywise correction such as Bonferroni or Holm guarantees that, within a chosen set of tests, the chance of even one false positive stays at or below α. It says nothing about tests outside that set. So the real decision is which tests go in the same set, and the answer comes from your conclusions, not from your spreadsheet: a family is the group of tests whose results jointly feed a single claim, so that one false positive would make that claim wrong.

• One family per outcome is reasonable when each outcome is a separate research question with its own conclusion, for example "does training improve recognition?" and, separately, "does it improve recall?". Each conclusion is then protected at α.
• One family for everything is needed when you will claim something general, such as "condition 1 beats condition 2", on the strength of any of the outcomes. Any false positive anywhere would then support that claim, so all the tests must share one correction.
• Fix the families in advance (ideally in a pre-registration or analysis plan) and state them in the paper. Choosing the grouping after seeing which version leaves more results significant defeats the purpose.

This page is about how to draw the boundary of a family. For the separate question of whether to correct at all, and how Bonferroni, Holm and Benjamini-Hochberg compare, see when to use a Bonferroni correction.

See it in R and Python

The code has two parts. Part 1 takes nine p-values from a design with three pairwise comparisons on each of three outcomes and applies Holm two ways: within each outcome, and to all nine together. Part 2 simulates 2,000 studies with three groups of 30 and three outcomes that correlate at .5, where every null hypothesis is true. For each rule it records how often a study produced at least one false positive within a single outcome, and anywhere among the nine tests.

set.seed(677506)

# 1. Nine p-values: 3 pairwise comparisons x 3 outcomes (no randomness)
p <- c(0.004, 0.030, 0.210,   # outcome A
       0.012, 0.048, 0.600,   # outcome B
       0.020, 0.090, 0.330)   # outcome C
outcome <- rep(c("A", "B", "C"), each = 3)
tab <- cbind(raw         = p,
             per_outcome = ave(p, outcome, FUN = function(x) p.adjust(x, "holm")),
             all_nine    = p.adjust(p, "holm"))
round(tab, 3)
colSums(tab < 0.05)   # tests significant at .05

# 2. Simulation: 3 groups (n = 30 each), 3 outcomes correlated at .5,
#    every null hypothesis true, so every rejection is a false positive
n <- 30; rho <- 0.5; reps <- 2000
g <- rep(1:3, each = n)
pairs <- combn(3, 2)
res <- replicate(reps, {
  z <- rnorm(3 * n)                     # shared person-level component
  Y <- sapply(1:3, function(k) sqrt(rho) * z + sqrt(1 - rho) * rnorm(3 * n))
  p <- as.vector(apply(Y, 2, function(y) apply(pairs, 2, function(pr)
         t.test(y[g == pr[1]], y[g == pr[2]], var.equal = TRUE)$p.value)))
  out <- rep(1:3, each = 3)
  rej <- list(none        = p < 0.05,
              per_outcome = ave(p, out, FUN = function(x) p.adjust(x, "holm")) < 0.05,
              all_nine    = p.adjust(p, "holm") < 0.05)
  sapply(rej, function(r) c(within_one_outcome = any(r[out == 1]),
                            anywhere           = any(r)))
})
round(apply(res, c(1, 2), mean), 3)   # rates of at least one false positive

import numpy as np
from scipy import stats
from itertools import combinations

rng = np.random.default_rng(677506)

# Holm adjustment, same definition as R's p.adjust(p, "holm")
def holm(p):
    p = np.asarray(p, float); m = len(p); o = np.argsort(p)
    out = np.empty(m)
    out[o] = np.minimum(1, np.maximum.accumulate((m - np.arange(m)) * p[o]))
    return out

def per_outcome_holm(p, k=3):
    return np.concatenate([holm(p[i:i + k]) for i in range(0, len(p), k)])

# 1. Nine p-values: 3 pairwise comparisons x 3 outcomes (no randomness)
p = np.array([0.004, 0.030, 0.210,    # outcome A
              0.012, 0.048, 0.600,    # outcome B
              0.020, 0.090, 0.330])   # outcome C
tab = np.column_stack([p, per_outcome_holm(p), holm(p)])
print("raw  per_outcome  all_nine")
print(np.round(tab, 3))
print("significant at .05:", (tab < 0.05).sum(axis=0))

# 2. Simulation: 3 groups (n = 30 each), 3 outcomes correlated at .5,
#    every null hypothesis true, so every rejection is a false positive
n, rho, reps = 30, 0.5, 2000
z = rng.normal(size=(reps, 1, 3 * n))               # shared person-level component
Y = np.sqrt(rho) * z + np.sqrt(1 - rho) * rng.normal(size=(reps, 3, 3 * n))
groups = [Y[:, :, i * n:(i + 1) * n] for i in range(3)]
# p-values ordered as in R: outcome 1 (pairs 12, 13, 23), outcome 2, outcome 3
pv = np.stack([stats.ttest_ind(groups[a], groups[b], axis=2).pvalue
               for a, b in combinations(range(3), 2)], axis=2).reshape(reps, 9)
rules = {"none": lambda r: r < 0.05,
         "per_outcome": lambda r: per_outcome_holm(r) < 0.05,
         "all_nine": lambda r: holm(r) < 0.05}
for name, rule in rules.items():
    rej = np.array([rule(row) for row in pv])
    print(name, "within_one_outcome", round(float(rej[:, :3].any(axis=1).mean()), 3),
          "anywhere", round(float(rej.any(axis=1).mean()), 3))

        raw per_outcome all_nine
 [1,] 0.004       0.012    0.036
 [2,] 0.030       0.060    0.180
 [3,] 0.210       0.210    0.630
 [4,] 0.012       0.036    0.096
 [5,] 0.048       0.096    0.240
 [6,] 0.600       0.600    0.660
 [7,] 0.020       0.060    0.140
 [8,] 0.090       0.180    0.360
 [9,] 0.330       0.330    0.660
        raw per_outcome    all_nine 
          5           2           1 
                    none per_outcome all_nine
within_one_outcome 0.138       0.052    0.017
anywhere           0.307       0.126    0.040

The simulated rates come from one seeded run of the R code. Python's random numbers differ from R's, so its simulated rates differ slightly (for example 12.2% instead of 12.6% for a false positive anywhere under per-outcome Holm), but they show the same pattern. Part 1 involves no randomness, and Python reproduces those adjusted p-values exactly.

What the numbers show

• The grouping changes the verdicts. Five of the nine raw p-values are below .05. Holm within each outcome keeps two of them (adjusted p = .012 for outcome A and .036 for outcome B); Holm across all nine keeps only one (adjusted p = .036). Neither answer is wrong; they protect different claims.
• Per-outcome correction protects each outcome. With no correction, 13.8% of simulated studies had a false positive within a given outcome. Holm per outcome brought this to 5.2%, which is within simulation noise of the 5% target (the Monte Carlo standard error is about 0.5 percentage points).
• But it does not protect the study as a whole. Under per-outcome Holm, 12.6% of studies had a false positive somewhere among the nine tests. With three families each held at 5%, the overall rate can reach roughly 3 × 5% = 15%; the correlation between outcomes kept it a little lower here. With no correction at all it was 30.7%.
• Pooling protects the overall claim, at a price. One Holm correction over all nine tests held the study-wide rate to 4.0%. Because the outcomes are correlated, the correction is somewhat conservative, and within a single outcome the error rate fell to 1.7%, which means less power to detect real differences.

Questions that settle the grouping

• What sentence will the abstract contain? If it says "method X improved vocabulary" and any of the outcomes could earn that sentence, the outcomes form one family. If it reports each outcome as its own finding, with its own interpretation, separate families are defensible.
• Would a reader accept a win on one outcome as a win overall? If yes, pool. Splitting families while drawing a pooled conclusion gives that conclusion an error rate several times α without saying so.
• Is one outcome primary? Then a hierarchy often beats both options. Test the primary outcome at the full α, and move on to secondary outcomes only if it is significant (a fixed-sequence or gatekeeping procedure). This controls the overall familywise error rate without splitting α, as long as the order is fixed in advance.
• Are the tests confirmatory or exploratory? Confirmatory tests that support the paper's main claims need error control as a group; clinical trials, for example, are usually expected to control it across all primary and key secondary endpoints. Clearly labelled exploratory analyses can be reported without a familywise correction, or with false discovery rate control.
• Are the outcomes strongly correlated? That is not a reason to skip a correction, but it does mean Bonferroni and Holm over-correct. Resampling-based methods, or a multivariate model that tests all outcomes at once, can recover some of the lost power.

Whatever you decide, write it down before looking at the results, apply the same rule to every outcome, and report every test in the family. If you also need to choose between planned and post hoc comparisons, see planned contrasts vs post hoc tests; our test chooser and the DASS blog cover choosing and reporting tests more broadly.

How to report adjusted p-values in APA style (7th edition)

State the families, the correction and the number of tests in each family in the method section, then give adjusted p-values. Using the per-outcome result from the example above:

"Pairwise comparisons among the three groups were adjusted with the Holm method separately for each outcome (three families of three comparisons), because each outcome addressed a distinct research question. For outcome A, one comparison remained significant (adjusted p = .012), as did one comparison for outcome B (adjusted p = .036); no comparison for outcome C was significant (all adjusted ps ≥ .060)."

If you pooled the tests instead: "To control the familywise error rate across all nine comparisons, p-values were adjusted with the Holm method; one comparison remained significant (adjusted p = .036)." In a table, list raw and adjusted p-values side by side and name the method and family size in a table note.

Related tools and guides

• Check a reported p-value
• Bonferroni correction: when should you use it, and what are the alternatives?
• Planned contrasts vs post hoc tests
• How are alpha and beta (Type I and Type II error rates) related?
• Which statistical test should I use?
• Family-wise error rate (Wikipedia)

More answered questions

• Planned contrasts vs post hoc tests: why do they give different p-values?
• Bonferroni correction: when should you use it, and what are the alternatives?
• Bayesian vs frequentist statistics in plain English

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.