Answer

How do you compare two classifiers across several datasets?

Inspired by a question on Cross Validated · November 9, 2026

classification nonparametric hypothesis testing

The short answer

Treat each dataset as one paired observation: get one score per method per dataset, then analyse the differences. A win count uses only the direction, a mean uses the size but a few datasets can sway it, and the Wilcoxon signed-rank test sits between the two. Report a summary of the differences and name the datasets where a method failed, because no single number shows both.

Treat each dataset as one observation

Suppose two methods have each been run on the same collection of datasets, and you have a score such as AUPR for every method on every dataset. The natural unit of analysis is the dataset, and the design is paired: both methods faced the same data. So the working quantity is one difference per dataset, method B's score minus method A's. See paired vs independent t test for why pairing matters.

Two consequences follow. Your sample size is the number of datasets, not the number of rows or cross-validation folds. Folds from one dataset share training data, so their results are correlated; average them into one score per dataset first. And whatever you conclude applies to datasets like the ones you tested. If they all came from one generator with different settings, you have learned about that family of problems and not about classification in general.

What each way of combining the scores tells you

Mean score or mean difference. Answers how much you gain on average. It uses the size of every result, and one or two extreme datasets can move it a long way. It also assumes that 0.02 means about the same thing on every dataset.
Median difference. Describes the typical dataset and ignores how badly things went in the tails.
Win count (sign test). Uses only which method came out ahead. It is the most robust option and the least powerful, because a win by 0.001 counts the same as a win by 0.20.
Wilcoxon signed-rank test. Ranks the absolute differences, so bigger gaps count for more, but an extreme gap cannot count for more than its rank. It needs differences that can be meaningfully compared in size across datasets. Its assumptions are worth a read.
Worst-case summary. The smallest difference, or a list of the datasets where a method failed. If you care about reliability, this may matter more than any average.

These describe different things, so choose according to the decision you need to make, and do it before looking at the results. For comparing classifiers, Demšar (2006, Journal of Machine Learning Research) recommended the Wilcoxon signed-rank test for two methods because it is safer than a t test when differences are not normal and more powerful than a plain win count.

Is it fine to average a score across datasets?

Averaging needs no special justification as a description: a macro-averaged Jaccard index is simply the mean of the per-dataset Jaccard values, and it treats every dataset as equally important. The alternative is micro-averaging, where you add up the intersections and the unions over all datasets before dividing, so large datasets dominate. Both are legitimate, and they answer different questions. Pick macro if each dataset is a separate problem you care about equally, and micro if you care about the items pooled together.

The real risk is mixing scores that are not on a common scale. The AUPR of a random classifier equals the proportion of positives, so 0.60 is excellent when positives are 2% of the data and poor when they are 55%. Before averaging, consider subtracting the baseline or ranking each method against random results on that dataset. The adjusted Rand index already has an expected value of 0 for random labelings; the Jaccard index has no such correction.

Avoid collapsing unrelated metrics into one composite number unless you can defend the weights. Choose a primary metric in advance, report the others alongside it, and correct for multiple comparisons if you test several.

See it in R and Python

The table below uses 18 invented datasets. Method B is slightly better on most of them and fails badly on three. The data are fixed, with no random numbers, so the figures quoted are the exact output of the R code and the Python version prints the same values.

# AUPR of two methods on the same 18 datasets (one value per dataset)
aupr_a <- c(.820, .740, .680, .610, .770, .580, .710, .850, .660,
            .630, .790, .570, .720, .880, .690, .750, .600, .810)
aupr_b <- c(.699, .642, .596, .628, .801, .592, .736, .859, .653,
            .652, .805, .604, .741, .869, .718, .756, .619, .834)

d <- aupr_b - aupr_a   # the paired difference is the unit of analysis
n <- length(d)

# Different summaries of the same 18 differences
round(c(mean_A = mean(aupr_a), mean_B = mean(aupr_b),
        mean_diff = mean(d), median_diff = median(d)), 4)
round(sort(d)[1:3], 3)          # the three worst results for B

t.test(d)                       # paired t test: is the mean difference 0?

wins <- sum(d > 0)
wins
round(binom.test(wins, n)$p.value, 4)   # sign test: does B win more than half?

wilcox.test(aupr_b, aupr_a, paired = TRUE)   # signed-rank test

# Matched-pairs rank-biserial correlation as an effect size
ranks <- rank(abs(d))
round((sum(ranks[d > 0]) - sum(ranks[d < 0])) / sum(ranks), 3)

Python

import numpy as np
from scipy import stats

# AUPR of two methods on the same 18 datasets (one value per dataset)
aupr_a = np.array([.820, .740, .680, .610, .770, .580, .710, .850, .660,
                   .630, .790, .570, .720, .880, .690, .750, .600, .810])
aupr_b = np.array([.699, .642, .596, .628, .801, .592, .736, .859, .653,
                   .652, .805, .604, .741, .869, .718, .756, .619, .834])

d = aupr_b - aupr_a   # the paired difference is the unit of analysis
n = len(d)

# Different summaries of the same 18 differences
print("mean A", round(aupr_a.mean(), 4), "mean B", round(aupr_b.mean(), 4))
print("mean diff", round(d.mean(), 4), "median diff", round(np.median(d), 4))
print("worst for B", np.round(np.sort(d)[:3], 3))

res = stats.ttest_1samp(d, 0)     # paired t test: is the mean difference 0?
ci = res.confidence_interval(0.95)
print("t", round(res.statistic, 3), "df", res.df, "p", round(res.pvalue, 4),
      "CI", round(ci.low, 4), round(ci.high, 4))

wins = int((d > 0).sum())
print("wins", wins, "sign test p", round(stats.binomtest(wins, n).pvalue, 4))

# Signed-rank test; V is the sum of the positive ranks, as in R
ranks = stats.rankdata(np.abs(d))
print("V", ranks[d > 0].sum(), "p", round(stats.wilcoxon(d, method="exact").pvalue, 4))

# Matched-pairs rank-biserial correlation as an effect size
print("rank-biserial", round((ranks[d > 0].sum() - ranks[d < 0].sum()) / ranks.sum(), 3))

Means. A averages 0.7144 and B 0.7113, a mean difference of -0.0031 (95% CI [-0.0265, 0.0203]; t(17) = -0.28, p = .782). Judged by the mean, the methods are tied.
Median. The median difference is +0.0165, so on a typical dataset B is ahead.
Wins. B wins on 13 of 18 datasets. The sign test gives p = .096.
Signed ranks. V = 114, exact p = .229, with a matched-pairs rank-biserial correlation of .33.
The failures. B's three worst differences are -0.121, -0.098 and -0.084. Those three cancel out its small gains in the mean.

None of the tests is significant, which does not show that the methods are equivalent: with 18 datasets the interval for the mean difference still includes B's typical gain of 0.0165. The useful finding is the pattern: small gains on most datasets and large losses on a few. The next step is to find out what those three datasets have in common, which no single aggregated number would have prompted.

More than two methods, or several metrics

For three or more methods on the same datasets, Demšar suggested the Friedman test followed by post hoc comparisons of mean ranks. Benavoli, Corani and Mangili (2016) later pointed out that a mean-rank comparison of two methods can change depending on which other methods are in the pool. A safer default is pairwise signed-rank tests on the methods you care about, with a Holm correction.

Before relying on any of this, check that the score you are comparing is the right one; why accuracy is a poor measure for classifiers covers common pitfalls. The test chooser can help if your design differs, and the DASS blog has more plain-language guides on reporting analyses.

How to report a classifier comparison in APA style (7th edition)

Say what the unit of analysis was, give an estimate with its interval, then the test and an effect size. Using the example above:

"We compared the two methods on 18 datasets, using one AUPR value per method per dataset. Method B scored higher on 13 datasets (Mdn difference = .0165), but the mean difference was close to zero (M = -.0031, 95% CI [-.0265, .0203]). A Wilcoxon signed-rank test did not indicate a statistically significant difference, V = 114, p = .229, rank-biserial r = .33. Method B performed markedly worse than Method A on three datasets."

A table with one row per dataset and the two scores, or a dot plot of the differences, lets readers see the failures for themselves. In the method section, state how each per-dataset score was computed, that the exact test was used, and any correction applied for testing several metrics or methods.

Related tools and guides

Working with your own data?

General answers only go so far. Send us your situation and we'll reply by email within two business days.

Ask your question

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.