Answer

What does Cohen's kappa tell you? A plain-language guide

Inspired by a question on Cross Validated ·

reliabilityclassification

The short answer

Cohen's kappa compares the agreement you observed with the agreement you would expect if two raters (or a classifier and the truth) were matched by chance alone. A kappa of 1 is perfect agreement, 0 is no better than chance, and a value such as .40 means you have closed 40% of the gap between chance-level and perfect agreement.

What kappa measures

Suppose two people sort the same set of items into categories, or a model's predictions are compared with the true labels. The obvious summary is percent agreement: how often the two match. The problem is that some matches happen by luck. If both raters say "no" most of the time, they will agree often even if they are not paying attention.

Cohen's kappa corrects for that. It takes the observed agreement, po, and the agreement expected by chance, pe, and asks how much of the room above chance was actually used: kappa = (po - pe) / (1 - pe). Chance agreement is worked out from each rater's own category rates, as if the two were assigning labels independently of each other.

How to read a kappa value

Many papers cite the Landis and Koch (1977) labels: 0-.20 slight, .21-.40 fair, .41-.60 moderate, .61-.80 substantial and .81-1 almost perfect. Treat these as rough conventions, not tests. What counts as good enough depends on the stakes: a screening tool used in research may live with .60, while a diagnosis that drives treatment needs far more.

A worked example in R and Python

The code builds two small tables by hand and computes kappa with a simple large-sample standard error. There is no randomness, so R and Python print the same figures. In practice, irr::kappa2() or psych::cohen.kappa() in R and sklearn.metrics.cohen_kappa_score() in Python do the same calculation from raw labels.

R

# Two raters sort the same 100 items into three categories
# (rows = rater 1, columns = rater 2)
tab <- matrix(c(30,  5,  2,
                 4, 35,  3,
                 3,  4, 14), nrow = 3, byrow = TRUE,
              dimnames = list(rater1 = c("yes", "no", "unsure"),
                              rater2 = c("yes", "no", "unsure")))

kappa_stats <- function(tab) {
  n  <- sum(tab)
  po <- sum(diag(tab)) / n                         # observed agreement
  pe <- sum(rowSums(tab) * colSums(tab)) / n^2     # agreement expected by chance
  k  <- (po - pe) / (1 - pe)
  se <- sqrt(po * (1 - po) / (n * (1 - pe)^2))     # simple large-sample SE
  round(c(po = po, pe = pe, kappa = k,
          lower = k - 1.96 * se, upper = k + 1.96 * se), 3)
}
kappa_stats(tab)

# A classifier on imbalanced data: 950 negatives, 50 positives
# (rows = actual, columns = predicted; order: negative, positive)
lazy <- matrix(c(950, 0, 50, 0), nrow = 2, byrow = TRUE) # always predicts "negative"
kappa_stats(lazy)     # accuracy .95, kappa 0

useful <- matrix(c(930, 20, 20, 30), nrow = 2, byrow = TRUE) # finds 30 of the 50 positives
kappa_stats(useful)   # accuracy .96, kappa well above 0

Python

import numpy as np

# Two raters sort the same 100 items into three categories
# (rows = rater 1, columns = rater 2; order: yes, no, unsure)
tab = np.array([[30,  5,  2],
                [ 4, 35,  3],
                [ 3,  4, 14]])

def kappa_stats(tab):
    tab = np.asarray(tab, dtype=float)
    n = tab.sum()
    po = np.trace(tab) / n                               # observed agreement
    pe = np.sum(tab.sum(axis=1) * tab.sum(axis=0)) / n**2  # agreement expected by chance
    k = (po - pe) / (1 - pe)
    se = np.sqrt(po * (1 - po) / (n * (1 - pe) ** 2))     # simple large-sample SE
    vals = {"po": po, "pe": pe, "kappa": k,
            "lower": k - 1.96 * se, "upper": k + 1.96 * se}
    return {name: round(float(v), 3) for name, v in vals.items()}

print(kappa_stats(tab))

# A classifier on imbalanced data: 950 negatives, 50 positives
# (rows = actual, columns = predicted; order: negative, positive)
lazy = [[950, 0], [50, 0]]        # always predicts "negative"
print(kappa_stats(lazy))          # accuracy .95, kappa 0

useful = [[930, 20], [20, 30]]    # finds 30 of the 50 positives
print(kappa_stats(useful))        # accuracy .96, kappa well above 0
   po    pe kappa lower upper 
0.790 0.362 0.671 0.546 0.796 
   po    pe kappa lower upper 
 0.95  0.95  0.00 -0.27  0.27 
   po    pe kappa lower upper 
0.960 0.905 0.579 0.451 0.707

In the first table the raters agree on 79 of 100 items. Chance alone would give about 36% agreement, so kappa is .671, with an approximate 95% interval from .546 to .796. On the Landis and Koch scale that is substantial agreement, though the interval reaches down into moderate.

Why kappa helps when judging a classifier

The second part of the example shows why kappa is popular for classifiers. With 950 negatives and 50 positives, a model that always predicts "negative" is 95% accurate, yet its kappa is exactly 0: it agrees with the truth only as often as its label rates guarantee. A model that finds 30 of the 50 positives, at the cost of 20 false alarms, is only one point more accurate (96%) but has a kappa of .579.

So kappa answers a sharper question than accuracy: how much better is this model than one that just reproduces the class frequencies? That makes it useful when classes are imbalanced. It is still a single number, so look at the confusion matrix, and at precision and recall when the costs of the two errors differ. See why accuracy is a poor measure for classifiers for more on that.

Limits and variants

Use the test chooser if you are unsure whether kappa or a correlation-type measure fits your data.

How to report Cohen's kappa in APA style (7th edition)

Give the number of items, the raw agreement, kappa with its confidence interval, and your interpretation. Kappa cannot exceed 1, so drop the leading zero. Using the first table above:

"Two raters independently coded 100 responses. They agreed on 79% of responses, Cohen's κ = .67, 95% CI [.55, .80], indicating substantial agreement (Landis & Koch, 1977)."

For a classifier, report kappa alongside accuracy, for example "accuracy = .96, κ = .58, 95% CI [.45, .71]", and say in the method section how the confidence interval was computed.

Related tools and guides

More answered questions

Working with your own data?

General answers only go so far. Send us your situation and we'll reply by email within two business days.

Ask your question

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.