Expected Calibration Error (ECE)

Summary

Expected Calibration Error measures how well a model’s predicted probabilities match empirical accuracies. Bin predictions by predicted confidence, and compute the weighted average of $∣ accuracy in bin - average confidence in bin ∣$ :

ECE = m = 1 \sum M \frac{∣ B _{m} ∣}{N} \cdot acc (B_{m}) - conf (B_{m}) .

A classifier that scores well on accuracy can still produce wildly miscalibrated probabilities. Predicting “90% confident” when only 60% of such predictions are correct. Calibration matters whenever:

The probability is used downstream (decision thresholds, expected-cost calculations, risk scoring).
A human reads the probability (medical diagnosis, fraud alerts).
The model is combined with other signals (Bayesian fusion).

ECE is the standard single-number calibration metric.

The mechanism

For a classifier producing a predicted label $\overset{y}{^}_{i}$ and its confidence $c_{i} \in [0, 1]$ on each of $N$ examples with true labels $y_{i}$ :

Bin the predictions by confidence. Standard: $M = 10$ equal-width bins covering $[0, 1]$ .
For each bin $B_{m}$ :
- $conf (B_{m}) = \frac{1}{∣ B _{m} ∣} \sum_{i \in B_{m}} c_{i}$ (average predicted-class confidence).
- $acc (B_{m}) = \frac{1}{∣ B _{m} ∣} \sum_{i \in B_{m}} 1 [\overset{y}{^}_{i} = y_{i}]$ (empirical accuracy).
Aggregate: ECE = weighted average of bin gaps.

Sample ECE is zero when each populated bin’s empirical accuracy equals its average confidence. Finite-sample noise and binning mean this does not prove population calibration. Conversely, a calibrated model need not have zero empirical ECE on a finite sample. The worked example below is multiclass, so predicted-class confidence can be below 50%.

Worked aggregation

A big error in a small bin can match a small error in a large bin

Derivation

Read it this way: multiply across each row, then add down the ECE column. Bin C has the largest gap, but only 10% of examples, so it contributes the same four points as the denser Bin B. This original example follows the ECE definition in Naeini et al. (2015) and Guo et al. (2017).

Check: Why does C's 40-point gap contribute the same four ECE points as B's ten-point gap?

Read it this way: A 10% bin with a 40-point gap and a 40% bin with a 10-point gap both contribute four percentage points to ECE.
What happens if only the first bin doubles in size?

Reliability diagram

The visual companion to ECE: plot bin accuracy vs. bin confidence. Perfect calibration is the diagonal $y = x$ . Above-diagonal: under-confident. Below-diagonal: over-confident (the typical deep-net failure).

Always plot the reliability diagram alongside reporting ECE. Single ECE number can hide dramatic per-bin issues.

Variants

Maximum Calibration Error (MCE): $max_{m} ∣ acc (B_{m}) - conf (B_{m}) ∣$ . Worst-case bin gap.
Adaptive ECE: equal-frequency bins instead of equal-width. Stable when predictions concentrate near 0 or 1.
Class-wise ECE: per-class calibration; matters in multi-class.
Top-label ECE (multi-class): compute ECE on the predicted-class probability only.

Why deep nets are miscalibrated

Modern neural networks (Guo et al., 2017) are typically overconfident:

Guo et al. observed overconfidence in the classification models and datasets they studied, not every architecture or deployment population.
Cross-entropy is a proper probability objective; fitting it on finite training data does not guarantee calibrated deployment probabilities.
Regularization, model fitting, sample size, and distribution shift can affect calibration. Measure their effect rather than assuming an architecture ordering.

Calibration methods

Fit a post-hoc map on calibration data, then evaluate the frozen map on separate, representative cases. The calibration lesson derives the maps and their effects on downstream decisions.

Temperature scaling (Guo et al., 2017): divide every class logit by one learned scalar $T > 0$ . Values above one soften probabilities; values below one sharpen them. Shared positive scaling preserves multiclass argmax, not every probability-threshold action.
Platt scaling: fit a binary logistic map with a slope and intercept. A shifted intercept can change decisions at probability 0.5.
Isotonic regression: fit a nondecreasing map. Plateaus can create ties or move scores across an operating threshold; flexible maps need enough calibration data.
Vector / matrix scaling: use class-specific scales/biases or a full affine logit map. These can change class ordering and accuracy.

Calibration is not generally lossless on accuracy or decision cost. Evaluate probability quality and the actual decision rule, including relevant slices, after fitting the map.

Predict before revealing: logits [2,1] choose class 0. Which class wins after shared temperature 10, versus class-specific temperatures [10,1]?

Check the ordering and the action

Shared scaling gives [0.2,0.1], so class 0 still wins. Class-specific scaling gives [0.2,1], so class 1 wins. Even shared scaling changes probabilities, so a cost-based threshold can trigger a different action without changing argmax.

Limits of ECE

Binning artifact: ECE depends on bin choice. Adaptive ECE is more stable.
Confidence vs. probability of correctness: ECE on multi-class typically uses only the top-class probability, ignoring whether the full distribution is well-calibrated.
Doesn’t measure sharpness: a model that predicts $0.5$ for everything has ECE = 0 if base rate is $0.5$ , but is useless. Combine ECE with proper scoring rules (Brier score, log loss).

Common pitfalls

Fitting and evaluating on the same cases. Keep calibration fitting separate from final evaluation; retest after population changes.
Reporting ECE without a reliability diagram. Visual asymmetries can be invisible in the single number.
Assuming rescaling must help. Compare the frozen map with the uncalibrated baseline on held-out probability and decision metrics.
Confusing calibration with accuracy. A 95%-accuracy model with ECE 0.20 is still untrustworthy whenever the probability matters.

Calibration. Broader treatment of calibration methods.
Confusion matrix. Accuracy-based metrics.