Answer

Dunn's test after Kruskal-Wallis: which p-value adjustment?

Inspired by a question on Cross Validated · November 2, 2026

nonparametricmultiple comparisons

The short answer

The Kruskal-Wallis test tells you whether at least one of several groups tends to have higher or lower values than the others. To find out which groups differ, the usual follow-up is Dunn's test: it ranks all observations together once, then compares the mean ranks of each pair of groups with a z statistic. (The Conover-Iman test is a close relative that uses a t distribution and is usually a little more powerful; everything below applies to it too.)

• Running the pairs only after a significant Kruskal-Wallis result is not enough protection in general. It works with three groups, for the same reason Fisher's LSD works with three groups after ANOVA. With four or more groups, one very different group can make the omnibus test reject almost every time, and then the gate does nothing to protect the comparisons among groups that really are alike.
• So the number of groups matters, and with more than three you should adjust the pairwise p-values.
• Which adjustment depends on the error you want to control. For "the chance of even one false claim" (the familywise error rate), use Holm, or Holm-Šidák if you want a sliver more power. For "the expected share of false claims among the ones I make" (the false discovery rate), use Benjamini-Hochberg.

See it in R and Python

The code first builds one skewed dataset: four groups of 12 drawn from lognormal distributions, where A and B come from the same distribution and C and D are shifted upward. It runs kruskal.test(), computes Dunn's test by hand (with the usual correction for ties), and adjusts the six pairwise p-values five ways. It then simulates 4,000 studies with five groups of 12 in which A to D are identical and E sits far above them, and counts how often each rule wrongly declares a difference between two of A to D.

set.seed(677526)
k <- 4; n <- 12
g <- factor(rep(LETTERS[1:k], each = n))
y <- rlnorm(k * n, meanlog = rep(c(0, 0, 0.5, 1), each = n))   # skewed outcome
kruskal.test(y ~ g)

# Dunn's test: compare mean ranks from the joint ranking, with a tie correction
dunn <- function(y, g) {
  r <- rank(y); N <- length(y); ni <- table(g); rbar <- tapply(r, g, mean)
  ties <- table(r); s2 <- N * (N + 1) / 12 - sum(ties^3 - ties) / (12 * (N - 1))
  pr <- combn(levels(g), 2)
  z <- as.numeric((rbar[pr[1, ]] - rbar[pr[2, ]]) /
                  sqrt(s2 * (1 / ni[pr[1, ]] + 1 / ni[pr[2, ]])))
  data.frame(pair = paste(pr[1, ], pr[2, ], sep = "-"), z = z, p = 2 * pnorm(-abs(z)))
}
holm_sidak <- function(p) {               # step-down Sidak
  m <- length(p); o <- order(p)
  adj <- cummax(1 - (1 - p[o])^(m - seq_len(m) + 1)); adj[order(o)]
}
d <- dunn(y, g); m <- nrow(d)
d$bonf <- p.adjust(d$p, "bonferroni"); d$sidak <- 1 - (1 - d$p)^m
d$holm <- p.adjust(d$p, "holm"); d$holm_sidak <- holm_sidak(d$p)
d$BH <- p.adjust(d$p, "BH")
print(d, digits = 3)

# Does a significant Kruskal-Wallis test protect unadjusted follow-ups?
# Five groups: A-D identical, E far higher. Count false claims among A-D pairs.
sim <- replicate(4000, {
  g5 <- factor(rep(LETTERS[1:5], each = n))
  y5 <- rlnorm(5 * n, meanlog = rep(c(0, 0, 0, 0, 2), each = n))
  d5 <- dunn(y5, g5); null_pairs <- !grepl("E", d5$pair)
  kw <- kruskal.test(y5, g5)$p.value < 0.05
  c(kw_rejects = kw,
    gated_unadjusted = kw && any(d5$p[null_pairs] < 0.05),
    holm = any(p.adjust(d5$p, "holm")[null_pairs] < 0.05))
})
round(rowMeans(sim), 3)

import numpy as np
import pandas as pd
from itertools import combinations
from scipy import stats

rng = np.random.default_rng(677526)
k, n = 4, 12
g = np.repeat(list("ABCD"), n)
y = rng.lognormal(np.repeat([0, 0, 0.5, 1], n), 1)       # skewed outcome
print(stats.kruskal(*[y[g == lev] for lev in "ABCD"]))

# Dunn's test: compare mean ranks from the joint ranking, with a tie correction
def dunn(y, g):
    r = stats.rankdata(y); N = len(y); levels = sorted(set(g))
    _, t = np.unique(r, return_counts=True)
    s2 = N * (N + 1) / 12 - np.sum(t**3 - t) / (12 * (N - 1))
    rows = []
    for a, b in combinations(levels, 2):
        z = (r[g == a].mean() - r[g == b].mean()) / np.sqrt(
            s2 * (1 / np.sum(g == a) + 1 / np.sum(g == b)))
        rows.append((f"{a}-{b}", z, 2 * stats.norm.sf(abs(z))))
    return pd.DataFrame(rows, columns=["pair", "z", "p"])

def step(p, f, down=True):                # step-down (Holm-type) or step-up (BH)
    m = len(p); o = np.argsort(p); i = np.arange(1, m + 1)
    adj = f(p[o], m, i)
    adj = np.maximum.accumulate(adj) if down else np.minimum.accumulate(adj[::-1])[::-1]
    out = np.empty(m); out[o] = np.minimum(adj, 1); return out

d = dunn(y, g); p = d["p"].to_numpy(); m = len(p)
d["bonf"] = np.minimum(p * m, 1); d["sidak"] = 1 - (1 - p) ** m
d["holm"] = step(p, lambda q, m, i: q * (m - i + 1))
d["holm_sidak"] = step(p, lambda q, m, i: 1 - (1 - q) ** (m - i + 1))
d["BH"] = step(p, lambda q, m, i: q * m / i, down=False)
print(d.round(4))

# Does a significant Kruskal-Wallis test protect unadjusted follow-ups?
# Five groups: A-D identical, E far higher. Count false claims among A-D pairs.
g5 = np.repeat(list("ABCDE"), n); res = []
for _ in range(4000):
    y5 = rng.lognormal(np.repeat([0, 0, 0, 0, 2], n), 1)
    d5 = dunn(y5, g5); null_pairs = ~d5["pair"].str.contains("E").to_numpy()
    kw = stats.kruskal(*[y5[g5 == lev] for lev in "ABCDE"]).pvalue < 0.05
    holm = step(d5["p"].to_numpy(), lambda q, m, i: q * (m - i + 1))
    res.append([kw, kw and (d5["p"].to_numpy()[null_pairs] < 0.05).any(),
                (holm[null_pairs] < 0.05).any()])
print(dict(zip(["kw_rejects", "gated_unadjusted", "holm"], np.mean(res, axis=0).round(3))))

	Kruskal-Wallis rank sum test

data:  y by g
Kruskal-Wallis chi-squared = 11.348, df = 3, p-value = 0.009987

  pair      z       p   bonf  sidak   holm holm_sidak     BH
1  A-B  2.377 0.01747 0.1048 0.1004 0.0699     0.0681 0.0349
2  A-C -0.627 0.53069 1.0000 0.9893 1.0000     0.8966 0.7818
3  A-D -0.350 0.72639 1.0000 0.9996 1.0000     0.9251 0.7818
4  B-C -3.004 0.00267 0.0160 0.0159 0.0160     0.0159 0.0160
5  B-D -2.727 0.00640 0.0384 0.0378 0.0320     0.0316 0.0192
6  C-D  0.277 0.78176 1.0000 0.9999 1.0000     0.9251 0.7818
      kw_rejects gated_unadjusted             holm 
           0.998            0.107            0.008

All figures come from one seeded run of the R code. Python's random numbers differ from R's, so its single dataset comes out differently (its Kruskal-Wallis test is not significant), but its simulation shows the same pattern: 0.998, 0.113 and 0.007. The adjustment formulas in both versions match R's p.adjust().

Reading the six comparisons

The Kruskal-Wallis test rejects (H = 11.35 on 3 degrees of freedom, p = .010). Among the pairs, B ranks clearly below C (z = −3.00) and D (z = −2.73), and both survive every adjustment: Holm gives p = .016 and .032.

The interesting row is A versus B. Those two groups were simulated from the same distribution, yet the unadjusted p is .017. Bonferroni (.105), Šidák (.100), Holm (.070) and Holm-Šidák (.068) all decline to call it, while Benjamini-Hochberg (.035) does. That is not a bug in Benjamini-Hochberg: it allows a controlled share of false claims among the discoveries, and here one of its three discoveries is false. Notice also that the comparisons of A with C and D, which are real shifts in the simulation, are not detected by any method; with 12 observations per group, modest shifts often go unseen.

Does a significant Kruskal-Wallis result license unadjusted pairs?

The argument for skipping adjustment is that the omnibus test already stands guard: if all groups are alike, it rejects only 5% of the time, so the follow-ups rarely get the chance to mislead. That reasoning holds when every group is alike. It breaks when some groups differ and others do not.

In the simulation, group E is so far above the rest that the Kruskal-Wallis test rejected in 99.8% of studies. The gate was therefore almost always open, and the six unadjusted comparisons among the identical groups A to D produced at least one false claim in 10.7% of studies, more than twice the 5% people think they are getting. Holm-adjusted comparisons across all ten pairs made such a claim in only 0.8% of studies. (Dunn's test is somewhat conservative in this setting because its rank variance includes the distant group, so with unadjusted tests the problem would be worse with a less extreme layout or more groups.)

With exactly three groups the gate does work: if all three are alike it rejects only 5% of the time, and if only one pair is alike, there is only one comparison that can produce a false claim. With four or more groups, adjust. The same logic for ANOVA is laid out in our page on whether Tukey's HSD needs a significant F test first; the short version is that a properly adjusted method does not need the omnibus result at all.

Holm, Šidák or Benjamini-Hochberg?

• Holm controls the familywise error rate under any dependence between the tests and is always at least as powerful as Bonferroni, so there is no reason to use plain Bonferroni instead. It is the safe default; our page on the Bonferroni correction covers the family-size question.
• Šidák and Holm-Šidák replace α/m with 1 − (1 − α)^(1/m). They are valid for two-sided tests whose statistics are jointly normal, which Dunn's z statistics approximately are, even though pairs that share a group are correlated. The gain is tiny: in the example, .100 instead of .105 for A versus B. Use them if you like, but they will rarely change a conclusion.
• Benjamini-Hochberg answers a different question. It keeps the expected proportion of false claims among your significant results at 5%, not the chance of making any. That suits exploratory screening of many comparisons, where a few false leads are acceptable. If each pairwise claim will be read and cited on its own, readers usually assume familywise control, so Holm is the better match.

Whichever you choose, decide before looking at the results and say which one you used. For choosing the omnibus test itself, try our test chooser, and the DASS blog has more on rank-based tests.

How to report Dunn's test in APA style (7th edition)

In the method section, name the follow-up test and the adjustment: "Pairwise differences were tested with Dunn's test, with Holm's adjustment to control the familywise error rate at .05."

In the results, give the omnibus test first and then the adjusted comparisons, using the figures from the example above: "Scores differed across the four groups, H(3) = 11.35, p = .010. Dunn's tests with Holm's adjustment showed that group B ranked lower than group C, z = −3.00, p = .016, and group D, z = −2.73, p = .032; no other pairs differed (all adjusted ps > .06)." With many pairs, put the z values and adjusted p-values in a table and name the adjustment in a table note.

Related tools and guides

• APA 7 formatter for nonparametric tests
• Is an F-test gate needed before Tukey's HSD?
• Bonferroni correction: when should you use it?
• Planned contrasts vs post hoc tests
• Kruskal-Wallis test (Wikipedia)

More answered questions

• Is an F-test gate needed before Tukey's HSD?
• Pearson or Spearman correlation: which should you use?
• How many participants do you need when comparing several treatment arms?

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.