Epistemic vs aleatoric uncertainty

Predictive uncertainty separates within-model variation from disagreement only relative to the chosen inputs, likelihood and model distribution.

Summary

Which proposed collection action can reduce the predictive spread at this input? Fix input $x$ , observed dataset $D$ and a distribution over models indexed by $m$ . Use the hypothetical equal-weight mixture $Y ∣ m = 1 \sim N (0, 1)$ and $Y ∣ m = 2 \sim N (2, 1)$ . Each normal’s second parameter is its variance.

Quantity	Calculation	Result
Predictive mean	$(0 + 2) /2$	1
Average conditional variance	$(1 + 1) /2$	1
Variance of conditional means	$((0 - 1)^{2} + (2 - 1)^{2}) /2$	1
Total variance	$1 + 1$	2

Independently, $E [Y^{2}] = ((1 + 0^{2}) + (1 + 2^{2})) /2 = 3$ . Subtracting the squared mean gives $3 - 1 = 2$ again.

How they show up mathematically

Let $μ_{m} = E [Y ∣ x, m]$ and $μ = E_{m} [μ_{m}]$ ; assume finite second moments. Write $Y - μ = (Y - μ_{m}) + (μ_{m} - μ)$ and square. The cross term vanishes after conditioning on $m$ , because $E [Y - μ_{m} ∣ x, m] = 0$ . Thus, under the specified model mixture,

Var (Y ∣ x, D) = E_{m} [Var (Y ∣ x, m)] + Var_{m} (μ_{m}) .

Call these model-relative within and between terms aleatoric and epistemic, respectively. Both use squared output units. The identity does not require Gaussian components; their normal form only simplifies this example. More representative data can change model uncertainty, but no universal monotonic decrease is guaranteed.

Learning objective

Separated model means add uncertainty even when each model is confident

Derivation

Read it this way: lines show qualitative conditional ranges; diamonds show conditional means. Compare range width with mean separation. Equal endpoint envelopes do not imply equal variances or coverage. These are schematic model draws, not measured posterior adequacy. Original construction based on total variance and Kendall and Gal (2017).

Check: Which uncertainty term grows when model means separate while each model's predictive spread stays narrow?

Read it this way: Each model has two equally likely outcomes. Both lie 0.4 from that model mean. Within-model variance is 0.16. Between-model variance is 2.25. Their sum is 2.41.
Can individually concentrated models form a wide mixture?

Change the information or output

Add an informative sensor

New task: supply missing predictive information, rather than repeat the same observations.

Picture and change: suppose each conditional variance falls from one to $0.25$ , while means and mixture weights stay fixed.

Check: within variance becomes $0.25$ ; between variance remains one; total variance is $1.25$ . The old within-model noise was not irreducible relative to the expanded inputs.

Use when: the proposed sensor changes the conditioning information; the stated reduction still needs empirical validation.

Predict a binary label

New task: measure information spread for equally weighted probabilities $(0.9, 0.1)$ and $(0.1, 0.9)$ .

Picture and change: their mean is $(0.5, 0.5)$ . Use entropy in bits, not regression variance.

Check: predictive entropy is one bit; average conditional entropy is approximately $0.468996$ bits. The difference is approximately $0.531004$ bits of mutual information between the label and this mixture’s model index.

Use when: categorical predictions make an entropy decomposition appropriate. A diverse ensemble is not automatically an exact posterior.

Failure and transfer

All members can predict mean zero and variance $0.01$ while target outputs are near ten. Zero disagreement does not certify correctness, calibration or in-distribution inputs. Now change the Gaussian weights to three quarters and one quarter. What happens?

Reasoned answer

The predictive mean is $0.5$ . Between variance is $0.75$ , within variance remains one, and total variance is $1.75$ . Changing weights changes the distribution being averaged, not merely an uncertainty label.

Estimating each in practice

Aleatoric

For heteroskedastic Gaussian regression, predict mean $μ (x)$ and log variance $s (x)$ , making variance $e^{s (x)}$ positive. Ignoring the normalizing constant, NLL is $[s (x) + (y - μ (x))^{2} e^{- s (x)}] /2$ . Fit quality and likelihood assumptions determine whether this estimates actual conditional noise. See Kendall and Gal.

Epistemic

Approach	Approximation	Cost to assess
Deep ensemble	diversity across separately fitted predictors	multiple training and serving runs
MC dropout	stochastic masks in a compatible dropout-trained model	repeated forward passes
Variational, SWAG or Laplace	a specified approximate parameter distribution	fitting, storage and sampling overhead

Measure approximation adequacy instead of declaring a universal strongest method or minimum ensemble size. BatchNorm behavior must remain controlled when enabling dropout sampling. References: deep ensembles and MC dropout.

Decomposition

For discrete mixture weights $w_{m}$ , let $p = \sum_{m} w_{m} p_{m}$ . Then $H (p) = \sum_{m} w_{m} H (p_{m}) + I (Y; M)$ . The difference is model-index mutual information under that joint distribution, not an automatic measurement of real-world ignorance.

Where each matters in practice

Active learning weighs expected information or decision benefit against acquisition cost; noisy labels can still be useful. Abstention, fallback and exploration require task-specific action costs and target-domain validation. No decomposition provides a universal out-of-distribution detector or safety guarantee.

Common pitfalls

One softmax vector cannot identify how uncertainty divides across plausible models. Agreement can reflect common misspecification. A fair binary outcome has variance $0.25$ or entropy one bit; probability $0.5$ alone does not specify uncertainty units.