Answer

Precision-recall or ROC: which should you use for rare outcomes?

Inspired by a question on Cross Validated · November 2, 2026

classification

Two plots from the same confusion matrices

Both curves come from sweeping a cut-off across a model's scores and building a confusion matrix at each step. They just plot different ratios from that matrix:

• ROC curve: the true positive rate (recall, sensitivity) against the false positive rate (1 minus specificity). Both ratios are computed within a class: recall uses only the positives, the false positive rate uses only the negatives.
• Precision-recall curve: precision (the share of flagged cases that are truly positive, also called positive predictive value) against recall. Precision mixes the two classes, so it depends on how common positives are.

That one difference drives everything else. Precision can be written as recall × prevalence divided by (recall × prevalence + false positive rate × (1 − prevalence)). Hold the classifier fixed and make positives rarer, and the ROC curve stays where it is, while precision falls, because the same false positive rate now applies to a far larger pool of negatives.

When each one is the better choice

• Prefer ROC when both kinds of error matter on their own terms, when classes are reasonably balanced, or when you want a summary that carries over to populations with a different prevalence (the ROC curve is a property of the score, not of the case mix). Its area has a clean meaning: the chance that a random positive outranks a random negative.
• Prefer PR when positives are rare and the practical question is about the cases you act on: fraud alerts, screening referrals, retrieved documents, detected objects. Here true negatives are plentiful and uninteresting, and a false positive rate of 2% can still mean that most alerts are false. The PR curve puts that cost in plain view.
• Report both when the outcome is rare and readers may want either view, or when you compare models across data sets with different prevalence. They answer different questions, so showing both is not redundant.

A useful theoretical result (Davis and Goadrich, 2006) is that one curve dominates another in ROC space exactly when it dominates in PR space. So the two views cannot disagree about a model that is better everywhere. They can disagree when curves cross, because the PR curve gives far more weight to the high-score region where the rare positives must be found, while the ROC area spreads its weight over all cut-offs.

See the difference in R and Python

This simulation keeps the classifier's quality fixed (positives score 1.5 standard deviations higher than negatives on average) and changes only the prevalence: 1,000 positives and 1,000 negatives, then 100 positives and 9,900 negatives. It computes the ROC area from ranks, the average precision (the usual summary of the PR curve), and recall, false positive rate and precision at a cut-off of 1.5. The Python tab does the same with NumPy and checks the results against scikit-learn's roc_auc_score and average_precision_score.

set.seed(2718)
# Same classifier quality in both settings: positives score higher on average
make_data <- function(n_pos, n_neg) {
  data.frame(y = c(rep(1, n_pos), rep(0, n_neg)),
             score = c(rnorm(n_pos, mean = 1.5), rnorm(n_neg, mean = 0)))
}
balanced <- make_data(1000, 1000)   # prevalence 50%
rare     <- make_data(100, 9900)    # prevalence 1%

summarise <- function(d, cut = 1.5) {
  pos <- d$score[d$y == 1]; neg <- d$score[d$y == 0]
  n1 <- length(pos); n0 <- length(neg)
  # ROC AUC: share of positive-negative pairs ranked correctly
  r <- rank(d$score)
  auc <- (sum(r[d$y == 1]) - n1 * (n1 + 1) / 2) / (n1 * n0)
  # Average precision: mean precision at the rank of each positive
  o <- order(d$score, decreasing = TRUE)
  yy <- d$y[o]
  prec_at_k <- cumsum(yy) / seq_along(yy)
  ap <- mean(prec_at_k[yy == 1])
  # One working cut-off
  flagged <- d$score >= cut
  c(prevalence = n1 / (n1 + n0), roc_auc = auc, avg_precision = ap,
    recall = mean(pos >= cut), fpr = mean(neg >= cut),
    precision = sum(flagged & d$y == 1) / sum(flagged),
    n_flagged = sum(flagged))
}
round(rbind(balanced = summarise(balanced), rare = summarise(rare)), 3)

import numpy as np
import pandas as pd
from scipy.stats import rankdata
from sklearn.metrics import roc_auc_score, average_precision_score

rng = np.random.default_rng(2718)

# Same classifier quality in both settings: positives score higher on average
def make_data(n_pos, n_neg):
    y = np.r_[np.ones(n_pos), np.zeros(n_neg)]
    score = np.r_[rng.normal(1.5, 1, n_pos), rng.normal(0, 1, n_neg)]
    return y, score

def summarise(y, score, cut=1.5):
    pos, neg = score[y == 1], score[y == 0]
    n1, n0 = len(pos), len(neg)
    # ROC AUC: share of positive-negative pairs ranked correctly
    r = rankdata(score)
    auc = (r[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)
    # Average precision: mean precision at the rank of each positive
    yy = y[np.argsort(-score)]
    prec_at_k = np.cumsum(yy) / np.arange(1, len(yy) + 1)
    ap = prec_at_k[yy == 1].mean()
    # One working cut-off
    flagged = score >= cut
    return {"prevalence": n1 / (n1 + n0), "roc_auc": auc, "avg_precision": ap,
            "sklearn_auc": roc_auc_score(y, score),
            "sklearn_ap": average_precision_score(y, score),
            "recall": np.mean(pos >= cut), "fpr": np.mean(neg >= cut),
            "precision": (flagged & (y == 1)).sum() / flagged.sum(),
            "n_flagged": flagged.sum()}

balanced = summarise(*make_data(1000, 1000))   # prevalence 50%
rare = summarise(*make_data(100, 9900))        # prevalence 1%
print(pd.DataFrame([balanced, rare], index=["balanced", "rare"]).round(3).to_string())

         prevalence roc_auc avg_precision recall   fpr precision n_flagged
balanced       0.50   0.864          0.86  0.501 0.065     0.885       566
rare           0.01   0.849          0.10  0.450 0.070     0.061       741

The figures come from one seeded run of the R code. Python uses a different random number generator, so its figures differ slightly (for example, an ROC AUC of 0.839 in both settings and an average precision of 0.134 in the rare one), but the pattern is the same, and the hand-computed values match scikit-learn's exactly.

• ROC barely moves. The area is 0.864 with balanced classes and 0.849 with a 1% prevalence. The small gap is sampling noise; the true value is the same in both settings.
• Average precision collapses. It drops from 0.86 to 0.10. A model with no skill would score about the prevalence, so 0.10 is still ten times chance, but it is a sober number.
• Same cut-off, very different meaning. At a score of 1.5 the model catches about half the positives and flags about 7% of negatives in both settings. With balanced classes, 88.5% of the 566 flagged cases are real. With a 1% prevalence, only 45 of the 741 flagged cases are real, a precision of 0.061. The ROC curve alone would not warn you about that.

Practical points

• The baselines differ. A useless model follows the diagonal on a ROC plot (area 0.5) but a flat line at the prevalence on a PR plot. Always state the prevalence next to a PR curve or average precision, otherwise readers cannot tell good from bad.
• Do not join PR points with straight lines. Linear interpolation is fine in ROC space but overstates performance in PR space. Average precision avoids this by summing precision over the steps in recall, which is why it is preferred to a trapezoid-rule PR area.
• Multi-class problems. Both curves are defined for two classes, so draw one curve per class (that class against the rest) and summarise with a macro average (each class counts equally) or a micro average (pooled over all decisions, dominated by the common classes). Say which one you used.
• Neither curve checks calibration. Both depend only on how the scores rank cases. If the predicted probabilities themselves will be used, add a calibration plot or a proper scoring rule such as the Brier score or log loss, and consider decision curve analysis to judge the payoff at realistic cut-offs.

How to report average precision in APA style (7th edition)

Say in the method section what the scores were, whether performance was measured on a held-out set or by cross-validation, and how average precision was computed. Both measures cannot exceed 1, so write them without a leading zero, and always give the prevalence. Using the rare-outcome example above:

"In the evaluation sample (100 positive and 9,900 negative cases; prevalence 1%), the classifier achieved an area under the ROC curve of .85 and an average precision of .10 (chance level .01). At a cut-off of 1.5, recall was .45 and precision was .06."

Add confidence intervals in square brackets when you have them, for example from a bootstrap: AUC = .xx, 95% CI [.xx, .xx]. Show the ROC and PR curves side by side as a figure when classification performance is a main result.

Related tools and guides

• What is AUC? The area under the ROC curve, explained
• Why accuracy is a poor measure for classifiers
• Which statistical test should I use?
• Precision and recall (Wikipedia)

More answered questions

• Why Euclidean distance breaks down in high dimensions
• Why is accuracy a poor way to judge a classification model?
• What is AUC? The area under the ROC curve, explained

Written by AskStats with AI assistance. This is general information, not advice for your specific data or study. When the results matter, check your approach with a qualified statistician.