Answer
What does Cohen's kappa tell you? A plain-language guide
The short answer
Cohen's kappa compares the agreement you observed with the agreement you would expect if two raters (or a classifier and the truth) were matched by chance alone. A kappa of 1 is perfect agreement, 0 is no better than chance, and a value such as .40 means you have closed 40% of the gap between chance-level and perfect agreement.
What kappa measures
Suppose two people sort the same set of items into categories, or a model's predictions are compared with the true labels. The obvious summary is percent agreement: how often the two match. The problem is that some matches happen by luck. If both raters say "no" most of the time, they will agree often even if they are not paying attention.
Cohen's kappa corrects for that. It takes the observed agreement, po, and the agreement expected by chance, pe, and asks how much of the room above chance was actually used: kappa = (po - pe) / (1 - pe). Chance agreement is worked out from each rater's own category rates, as if the two were assigning labels independently of each other.
How to read a kappa value
- 1: perfect agreement on every item.
- 0: agreement no better than what the two sets of category rates would give by chance.
- Below 0: systematic disagreement, worse than chance. This is rare and usually signals a coding mix-up.
- In between: the share of the possible improvement over chance that was achieved. A kappa of .40 does not mean the raters agree 40% of the time; it means they closed 40% of the distance between chance-level and perfect agreement.
Many papers cite the Landis and Koch (1977) labels: 0-.20 slight, .21-.40 fair, .41-.60 moderate, .61-.80 substantial and .81-1 almost perfect. Treat these as rough conventions, not tests. What counts as good enough depends on the stakes: a screening tool used in research may live with .60, while a diagnosis that drives treatment needs far more.
A worked example in R and Python
The code builds two small tables by hand and computes kappa with a simple large-sample standard error. There is no randomness, so R and Python print the same figures. In practice, irr::kappa2() or psych::cohen.kappa() in R and sklearn.metrics.cohen_kappa_score() in Python do the same calculation from raw labels.
R
# Two raters sort the same 100 items into three categories
# (rows = rater 1, columns = rater 2)
tab <- matrix(c(30, 5, 2,
4, 35, 3,
3, 4, 14), nrow = 3, byrow = TRUE,
dimnames = list(rater1 = c("yes", "no", "unsure"),
rater2 = c("yes", "no", "unsure")))
kappa_stats <- function(tab) {
n <- sum(tab)
po <- sum(diag(tab)) / n # observed agreement
pe <- sum(rowSums(tab) * colSums(tab)) / n^2 # agreement expected by chance
k <- (po - pe) / (1 - pe)
se <- sqrt(po * (1 - po) / (n * (1 - pe)^2)) # simple large-sample SE
round(c(po = po, pe = pe, kappa = k,
lower = k - 1.96 * se, upper = k + 1.96 * se), 3)
}
kappa_stats(tab)
# A classifier on imbalanced data: 950 negatives, 50 positives
# (rows = actual, columns = predicted; order: negative, positive)
lazy <- matrix(c(950, 0, 50, 0), nrow = 2, byrow = TRUE) # always predicts "negative"
kappa_stats(lazy) # accuracy .95, kappa 0
useful <- matrix(c(930, 20, 20, 30), nrow = 2, byrow = TRUE) # finds 30 of the 50 positives
kappa_stats(useful) # accuracy .96, kappa well above 0Python
import numpy as np
# Two raters sort the same 100 items into three categories
# (rows = rater 1, columns = rater 2; order: yes, no, unsure)
tab = np.array([[30, 5, 2],
[ 4, 35, 3],
[ 3, 4, 14]])
def kappa_stats(tab):
tab = np.asarray(tab, dtype=float)
n = tab.sum()
po = np.trace(tab) / n # observed agreement
pe = np.sum(tab.sum(axis=1) * tab.sum(axis=0)) / n**2 # agreement expected by chance
k = (po - pe) / (1 - pe)
se = np.sqrt(po * (1 - po) / (n * (1 - pe) ** 2)) # simple large-sample SE
vals = {"po": po, "pe": pe, "kappa": k,
"lower": k - 1.96 * se, "upper": k + 1.96 * se}
return {name: round(float(v), 3) for name, v in vals.items()}
print(kappa_stats(tab))
# A classifier on imbalanced data: 950 negatives, 50 positives
# (rows = actual, columns = predicted; order: negative, positive)
lazy = [[950, 0], [50, 0]] # always predicts "negative"
print(kappa_stats(lazy)) # accuracy .95, kappa 0
useful = [[930, 20], [20, 30]] # finds 30 of the 50 positives
print(kappa_stats(useful)) # accuracy .96, kappa well above 0 po pe kappa lower upper
0.790 0.362 0.671 0.546 0.796
po pe kappa lower upper
0.95 0.95 0.00 -0.27 0.27
po pe kappa lower upper
0.960 0.905 0.579 0.451 0.707
In the first table the raters agree on 79 of 100 items. Chance alone would give about 36% agreement, so kappa is .671, with an approximate 95% interval from .546 to .796. On the Landis and Koch scale that is substantial agreement, though the interval reaches down into moderate.
Why kappa helps when judging a classifier
The second part of the example shows why kappa is popular for classifiers. With 950 negatives and 50 positives, a model that always predicts "negative" is 95% accurate, yet its kappa is exactly 0: it agrees with the truth only as often as its label rates guarantee. A model that finds 30 of the 50 positives, at the cost of 20 false alarms, is only one point more accurate (96%) but has a kappa of .579.
So kappa answers a sharper question than accuracy: how much better is this model than one that just reproduces the class frequencies? That makes it useful when classes are imbalanced. It is still a single number, so look at the confusion matrix, and at precision and recall when the costs of the two errors differ. See why accuracy is a poor measure for classifiers for more on that.
Limits and variants
- Prevalence matters. When one category is very common, chance agreement is high, so kappa can be low even when raw agreement is high. Report percent agreement and the category rates alongside kappa.
- Ordered categories. If the categories are ranked (mild, moderate, severe), use weighted kappa, which gives partial credit for near misses.
- More than two raters. Cohen's kappa compares exactly two. Fleiss' kappa or an intraclass correlation handles more.
- Small samples. With few items the interval is wide, so report it. The simple standard error used here is an approximation; packages may use a slightly different formula.
Use the test chooser if you are unsure whether kappa or a correlation-type measure fits your data.
How to report Cohen's kappa in APA style (7th edition)
Give the number of items, the raw agreement, kappa with its confidence interval, and your interpretation. Kappa cannot exceed 1, so drop the leading zero. Using the first table above:
"Two raters independently coded 100 responses. They agreed on 79% of responses, Cohen's κ = .67, 95% CI [.55, .80], indicating substantial agreement (Landis & Koch, 1977)."
For a classifier, report kappa alongside accuracy, for example "accuracy = .96, κ = .58, 95% CI [.45, .71]", and say in the method section how the confidence interval was computed.
Related tools and guides
- APA 7 formatter for reliability
- Why accuracy is a poor measure for classifiers
- Which statistical test should I use?
- Cohen's kappa (Wikipedia)
More answered questions
- How do you compare two classifiers across several datasets?
- Precision-recall or ROC: which should you use for rare outcomes?
- Why Euclidean distance breaks down in high dimensions
Working with your own data?
General answers only go so far. Send us your situation and we'll reply by email within two business days.
Ask your questionWritten by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.