ROC, PR curves, and AUC

Summary

For a binary classifier producing scores, the ROC curve plots true positive rate (TPR) vs. false positive rate (FPR) as the decision threshold sweeps from $- \infty$ to $+ \infty$ . The PR curve plots precision vs. recall over the same sweep. The area under each curve (AUC) is a single-number summary independent of any threshold choice.

A confusion matrix at one threshold tells you about that threshold; the curves and their AUCs summarize the classifier’s separability across all thresholds. Picking the wrong curve for your imbalance regime gives you a misleading optimism (ROC-AUC near 1 on heavy imbalance is common but uninformative).

The four base counts

	Predicted positive	Predicted negative
Actually positive	TP	FN
Actually negative	FP	TN

From these:

TPR = Recall = \frac{T P}{T P + F N}, FPR = \frac{F P}{F P + T N}, Precision = \frac{T P}{T P + F P} .

Spatial intuition

Rare positives let a small false-positive rate overwhelm true positives

Derivation

ROC coordinates

Rates are normalized within each true class.

Precision-recall coordinates

False alarms compete directly with true alerts.

with 10 positives and 1,000 negatives, 8 true alerts and 50 false alarms give 80% recall, 5% FPR, and only 14% precision. ROC hides the alert volume; PR exposes it.

Check: Why does five-percent FPR still produce low precision when only one percent of examples are positive?

Read it this way: Nine hits accompany 99 false alarms. Recall is 9/10, but precision is only 9/108, about 8.3%.
Why can an approximately one-percent false-positive rate still give low precision?

ROC curve and ROC-AUC

X-axis: FPR (false alarm rate among true negatives).
Y-axis: TPR (recall on positives).
Random classifier: diagonal from (0,0) to (1,1), AUC = 0.5.
Perfect classifier: top-left corner, AUC = 1.0.

Probabilistic interpretation: ROC-AUC is the probability that a random positive scores above a random negative, plus half the probability of a tie.

Changing only the class prior leaves ROC unchanged when the class-conditional score distributions stay fixed. It does not leave alert precision unchanged: at prevalence $π$ , precision is $π TPR / (π TPR + (1 - π) FPR)$ . A high ROC-AUC can therefore coexist with poor precision at the operating threshold; the AUC alone does not determine that threshold’s usefulness.

PR curve and PR-AUC

X-axis: Recall.
Y-axis: Precision.
Random-ranking reference: precision $π$ , the positive class prior. Finite-sample AP varies; this line is not a promise that every random ranking scores exactly $π$ .
Perfect classifier: top-right corner.

Choose a convention: average precision (AP) is the recall-increment-weighted sum $\sum_{n} (R_{n} - R_{n - 1}) P_{n}$ . Trapezoidal PR area instead linearly interpolates between successive precision-recall points. These are different estimators; report the metric/API and convention rather than using “PR-AUC” ambiguously. The scikit-learn AP definition explicitly makes this distinction.

Precision-recall metrics depend on prevalence and expose alert quality that ROC alone does not specify. Neither curve is inherently honest or dishonest; choose it for the decision, state the population, and report the operating point.

Which to report

Setting	Default metric
Balanced or near-balanced classification	ROC-AUC
Imbalanced (positives < 5%)	PR-AUC, plus precision@k
Search / retrieval / recsys	PR-AUC, recall@k, NDCG
Fraud / anomaly detection	PR-AUC, precision@high-recall point
Medical screening	PR-AUC + sensitivity at fixed specificity

For interview answers: if you give ROC-AUC alone for a fraud problem (1% positives), expect a follow-up.

Numerical example

Use descending scores [0.9,0.8,0.7] with labels [1,0,1]:

Accepted prefix	Recall	Precision
No predictions	0	1, by endpoint convention
First item	1/2	1
First two items	1/2	1/2
All three items	1	2/3

AP is $(1/2) \cdot 1 + (1/2) \cdot (2/3) = 5/6$ . Trapezoidal PR area is $1/2 + (1/2) (1/2 + 2/3) /2 = 19/24$ . The scores are identical; the integration rule changed.

Predict before revealing: another implementation returns about 0.7917 instead of 0.8333. Must its ranking be wrong?

Check the definition before the implementation

No. Those are the trapezoidal area and AP respectively for the same fixture. Specify the convention before comparing models or reproducing a reported metric. If all scores tie, ROC-AUC is 0.5 under the half-credit convention, not zero.

Relationship between the two

Concavity / shape: ROC is monotonically non-decreasing; PR can be jagged and non-monotonic.
Pareto-equivalent for ranking: a model dominates another in ROC iff it dominates in PR (Davis & Goadrich, 2006).
AUCs are not directly comparable across the two curves. Same model has very different ROC-AUC and PR-AUC values.

Common pitfalls

Grading a model from AUC and prevalence alone. Inspect precision/recall, false-alert volume, and decision costs at the operating threshold.
Comparing PR-AUC across datasets with different positive priors. They live on different baselines.
Treating high AUC as a deployment-ready signal. AUC measures separability across all thresholds; production needs one threshold, picked from a confusion matrix at the operating point.
Confusing ROC-AUC with “accuracy.” They are different quantities; accuracy is threshold-dependent, AUC is not.

Calibration. Predicted probabilities should match empirical frequencies.
Class imbalance. Handling imbalanced training data.