Confusion matrix and classification metrics

Summary

A confusion matrix counts predicted labels against actual labels. Rows and columns select different populations, so they answer different questions.

Suppose 1,000 held-out cases produce 98 alerts. Eight alerts are correct, and two positive cases receive no alert. How many alerts are useful? How many actual positives did the model find? Use mature labels and one fixed threshold for this example.

The binary confusion matrix

	Predicted 0	Predicted 1
Actual 0	TN (True Neg)	FP (False Pos, Type I)
Actual 1	FN (False Neg, Type II)	TP (True Pos)

Spatial intuition

Eight hits can mean high recall and low precision

Derivation

Precision: trust a positive prediction

Highlight the predicted-positive column.

Actual	Pred 0	Pred 1
0	900 TN	90 FP
1	2 FN	8 TP

precision = 8 / (8 + 90) ≈ 8.16%

Recall: find the actual positives

Highlight the actual-positive row.

Actual	Pred 0	Pred 1
0	900 TN	90 FP
1	2 FN	8 TP

recall = 8 / (8 + 2) = 80%

Read it this way: the same classifier has 80% recall and about 8.16% precision. The matrix shows many more false alarms than true alerts.

Check: How can the same eight true positives yield 80% recall but only about 8.16% precision?

Read it this way: Recall divides eight by ten actual positives. Precision divides the same eight by forty predicted positives.
Which extra cases enter precision but not recall?

Read the diagram’s positive prediction column: 90 false alerts plus eight true alerts. Precision is $8/98 \approx 0.08163$ , or about 8.16%. Read the actual-positive row: two misses plus eight true alerts. Recall is $8/10 = 0.8$ .

The remaining 900 cases are true negatives. Check the total: $900 + 90 + 2 + 8 = 1000$ . Accuracy is $908/1000 = 0.908$ , although most alerts are false.

The denominator must count the population named in the question. Let $N = T P + T N + F P + F N$ . These metrics use the four counts at this threshold:

Metric	Formula	Meaning
Accuracy	$(T P + T N) / N$	Overall correctness
Precision (PPV)	$T P / (T P + F P)$	Of predicted positives, fraction correct
Recall (TPR, Sensitivity)	$T P / (T P + F N)$	Of actual positives, fraction caught
Specificity (TNR)	$T N / (T N + F P)$	Of actual negatives, fraction correctly rejected
F1	$2 P R / (P + R)$	Harmonic mean of precision and recall
FPR	$F P / (F P + T N)$	False alarm rate; $1 - specificity$
FNR	$F N / (F N + T P)$	Miss rate; $1 - recall$
NPV	$T N / (T N + F N)$	Negative predictive value
Balanced accuracy	$(T P R + T N R) /2$	Average of recall on each class
Matthews Corr Coef	see below	Single-number summary that handles imbalance

Matthews Correlation Coefficient (MCC):

MCC = \frac{T P \cdot T N - F P \cdot F N}{( T P + F P ) ( T P + F N ) ( T N + F P ) ( T N + F N )} .

When its denominator is nonzero, MCC ranges from $- 1$ to $1$ . Perfect agreement gives $1$ ; complete reversal gives $- 1$ . Zero means no observed binary correlation, not proof of a useful or random model. MCC uses all four marginal totals. It is not invariant to class prevalence.

For this table, $F_{1} = 16/ (16 + 90 + 2) \approx 0.14815$ . Specificity is $900/990 \approx 0.90909$ . Balanced accuracy is $(0.8 + 0.90909) /2 \approx 0.85455$ . The table describes errors. Product costs determine whether those errors are acceptable.

Change the population

1. Find which actual class is missed

New task: compare recall across actual classes.

Picture and change: divide each row by its own total, rather than dividing a prediction column. Keep raw counts visible beside the normalized table.

Check: the actual-positive row becomes $[0.2, 0.8]$ . The positive prediction column instead gives precision $8/98$ . They describe different populations and need not agree.

Use when: the question concerns coverage of an actual class.

2. Give each class equal weight

New task: prevent the majority class from dominating a recall summary.

Picture and change: calculate recall for each class, then average the two results. Do not pool their counts first.

Check: equal weighting gives balanced accuracy, about $0.85455$ . Pooling correct predictions gives accuracy $0.908$ , because 990 of 1,000 cases are negative.

Use when: each class has equal decision importance, regardless of its frequency. Equal weighting still does not remove the need to inspect each class.

What this table cannot tell you

Two models can produce identical labels at threshold $0.5$ but order their scores differently. Their confusion matrices match, while ranking measures and log losses can differ. Use scores to evaluate those properties. One thresholded table cannot recover them.

No predicted positives gives precision denominator zero. No actual positives gives recall denominator zero. State the undefined-value convention and show the support. An assigned zero is a reporting policy, not an observed error rate.

Try a missing population

There are no positive predictions. Can precision prove that the model’s alerts are reliable?

Check your answer

No. There are no alerts to assess, so precision is undefined before applying a reporting convention. Report zero predicted support beside that convention.

Multi-class confusion matrix

For $K$ classes, use a $K \times K$ table. Diagonal entries are correct predictions; other entries are confusions. For class $k$ , treat all other classes as negative and recover counts from row and column totals.

Macro: average the per-class metrics equally.
Micro: combine counts first, then calculate the metric.
Weighted: weight each per-class metric by its support.

In ordinary single-label multiclass classification, micro precision and recall both equal accuracy. Do not extend that identity to multilabel predictions.

Visualizing the confusion matrix

Label both axes explicitly. Use row normalization for recall and column normalization for precision. Keep counts available and use the same class order on both axes.

Threshold dependence

Increasing the positive-score threshold cannot increase TP or FP on the same examples. Precision can still move either way. Threshold sweeps need the scores, not only one matrix. Choose the operating point on validation data using the stated costs or constraints.

What to report

Report counts, per-class metrics, the operating threshold and the aggregation convention. Add score-based curves when they answer the decision question. State whether PR area means average precision or another integration rule. Scikit-learn provides confusion_matrix, classification_report, roc_auc_score and average_precision_score.

Common pitfalls

State the axis convention; libraries differ.
Use held-out data, not training performance, for generalization claims.
Do not select a threshold on the final test set.
Inspect class supports. One average can hide a failed class.

Precision, recall, F1. Single-metric details.
ROC, PR, AUC. Threshold-free metrics.