Bayes' rule and the posterior

Summary

Bayes’ rule conditions a declared probability model on observed evidence. For hypothesis or parameter $θ$ , multiply its prior weight by its evidence likelihood, then normalize:

p (θ ∣ D) = \frac{p ( D ∣ θ ) p ( θ )}{p ( D )} \propto p (D ∣ θ) p (θ) .

The evidence normalizer must be positive and finite. For discrete hypotheses, sum their weighted likelihoods; continuous parameters require an integral under the declared measure. Posterior probabilities, a posterior mode, and a chosen action are different outputs.

The classic example

Consider a hypothetical test with sensitivity $P (+ ∣ d i se a se) = 0.99$ and specificity $P (- ∣ h e a l t h y) = 0.99$ . Assume prevalence $0.001$ and random sampling from that population; these are teaching assumptions, not measured clinical evidence. In $100000$ people, expected branch counts are $100$ diseased and $99900$ healthy. The branches contribute $99$ and $999$ positive results respectively, so the positive-result population has $1098$ members.

P (d i se a se ∣ +) = \frac{99}{99 + 999} = \frac{11}{122} \approx 0.090164.

Learning objective

Only 99 of 1,098 positive results come from the diseased group

Derivation

Read it this way: the narrow solid piece is 99 true positives. The much wider dashed piece is 999 false positives. Their widths use the same people-per-unit scale. The denominator is the entire strip, not just the diseased branch: 99 divided by 1,098, about 9%. These are expected counts under the hypothetical test assumptions.

Check: Why is a positive result only about 9% likely to indicate disease despite 99% sensitivity?

Read it this way: Among all 1,098 positive tests, only 99 are true positives. Sensitivity is not the answer to this question.
If disease prevalence rises, which input cohort changes before the test acts?

The healthy posterior is $999/1098$ ; together the two posterior masses sum to one.

The four pieces

Piece	Name	Role
$p (θ)$	Prior	Weight before this observation
$p (D ∣ θ)$	Likelihood	Evidence model at a candidate parameter
$p (θ ∣ D)$	Posterior	Normalized conditional distribution
$p (D)$	Evidence	Sum or integral of prior-weighted likelihoods

The evidence is constant across parameters within one model, so maximizing an unnormalized posterior can locate its mode. It cannot be discarded when reporting posterior probabilities or comparing model evidence. A constant prior density can preserve an MLE maximizer in that parameterization, when the posterior and maximizer exist. This is a prior assumption, not an absence of assumptions.

Change the Requirement

Change the Population

New task: Increase prevalence to one percent while keeping sensitivity and specificity unchanged.

Picture and change: Begin with $1000$ diseased and $99000$ healthy people.

Check: Both branches contribute $990$ positives; posterior disease probability becomes $1/2$ .

Use when: Screening and referral populations have different base rates and the transported test properties remain justified.

Observe Another Positive

New task: Add a second positive result, conditionally independent of the first given disease status.

Picture and change: Multiply each branch by its second-result likelihood.

Check: Original-population expected double-positive counts are $98.01$ and $9.99$ , giving $98.01/108 = 0.9075$ .

Use when: The observation supplies genuinely new conditional evidence. Copying the same test report is perfectly dependent evidence and leaves the original posterior unchanged.

Conjugate priors

A conjugate prior produces a posterior in the same family, with parameters updated from the observations.

Likelihood	Conjugate prior	Posterior family
Bernoulli/binomial	Beta	Beta
Categorical/multinomial	Dirichlet	Dirichlet
Gaussian (mean, known $σ^{2}$ )	Gaussian	Gaussian
Gaussian precision, known mean	Gamma	Gamma
Poisson	Gamma	Gamma

Used in: Thompson sampling for bandits (Beta-Bernoulli), online recsys updates, conjugate Gibbs samplers.

Approximate inference (when conjugacy fails)

When closed-form conditioning is unavailable, distinguish the approximation from the desired posterior:

Laplace: approximate a suitable interior mode with a Gaussian using local curvature.
Variational inference: optimize within a chosen distribution family; the family and divergence affect the approximation.
MCMC: construct a target-invariant chain; convergence and Monte Carlo uncertainty still require diagnosis.
Stochastic-gradient samplers: introduce data subsampling and step-size/approximation considerations.

Common pitfalls

Reversed conditioning. A likelihood does not need to normalize over parameters.
Prior sensitivity. Weak evidence can leave substantial sensitivity to the prior; small sample size alone does not prove prior dominance.
Mode-only reporting. A mode omits uncertainty and can change under reparameterization.
Improper priors. An unnormalized prior can produce either a proper or improper posterior; check the integral.

Reconstruct

Why cannot two copies of one positive report be treated as independent tests?

Once the first report is known, its exact copy is certain under either disease status. The copy supplies likelihood ratio one, not another factor of ninety-nine.