KL divergence: which distribution weights the error?

Summary

KL divergence averages a log probability ratio under its first distribution. Swapping the arguments changes that weighting, so the same candidates can receive opposite rankings. Start with expected surprisal, not a distance analogy.

Task: compare only two candidates for a three-outcome target:

p = (0.49, 0.02, 0.49), q_{co v er} = (0.34, 0.32, 0.34), q_{l e f t} = (0.96, 0.02, 0.02) .

Each vector sums to one. Use natural logarithms, giving scores in nats:

KL (p ∥ q) = E_{p} [lo g \frac{p ( X )}{q ( X )}] = x \sum p (x) lo g \frac{p ( x )}{q ( x )} .

Properties

Terms with $p (x) = 0$ contribute zero, including when $q (x) = 0$ . If $p (x) > 0$ but $q (x) = 0$ , the divergence is infinite. Otherwise finite probability vectors give a nonnegative score, zero exactly when the distributions agree. KL is not symmetric and does not satisfy metric guarantees such as the triangle inequality.

Forward vs. reverse KL

Hold the target and both candidates fixed while changing only the sampling weights.

Learning objective

Changing which distribution does the averaging reverses this candidate ranking

Derivation

Read it this way: hold the target and both candidates fixed, then compare the score rows. Forward KL weights errors by target mass; nearly missing the right mode is expensive. Reverse KL weights errors by candidate mass; the broad candidate pays for its extra middle mass. This is a restricted choice between two candidates, not a universal optimizer. Values are in nats, following Kullback and Leibler (1951). For divergence direction, see Minka (2005).

Check: Why does changing the expectation's weights reverse the preferred candidate without changing either candidate distribution?

Read it this way: Target P=(.9,.1): forward KL prefers Q1=(.7,.3); reverse KL prefers Q2=(.99,.01). Both panels use the same scale in nats; lower is better.
Which distribution supplies the averaging weights in reverse KL?

Connection to cross-entropy

Combining equal terms gives four complete expectations:

Expectation	Weighted log terms	Score, nats
$KL (p ∥ q_{co v er})$	$0.98 lo g (0.49/0.34) + 0.02 lo g (0.02/0.32)$	$0.302699$
$KL (p ∥ q_{l e f t})$	$0.49 lo g (0.49/0.96) + 0.49 lo g (0.49/0.02)$	$1.237811$
$KL (q_{co v er} ∥ p)$	$0.68 lo g (0.34/0.49) + 0.32 lo g 16$	$0.638716$
$KL (q_{l e f t} ∥ p)$	$0.96 lo g (0.96/0.49) + 0.02 lo g (0.02/0.49)$	$0.581653$

Forward KL prefers cover; reverse KL prefers left among these candidates. Reversing KL is not negating a forward score: it also changes the weights. For fixed discrete $p$ , add and subtract $lo g p$ :

H (p, q) = - x \sum p (x) lo g q (x) = - x \sum p (x) lo g p (x) + x \sum p (x) lo g \frac{p ( x )}{q ( x )} = H (p) + KL (p ∥ q) .

Only $H (p)$ stays constant when optimizing $q$ . Categorical log-loss minimization and forward-KL minimization therefore have the same optimizer over the chosen model family.

Remove candidate support

New task: replace the left candidate with $(0.98, 0.02, 0)$ . Picture and change: the target’s final outcome still has mass $0.49$ , but the candidate now assigns zero. Check: forward KL becomes infinite; reverse KL remains $0.98 lo g 2 \approx 0.679284$ nats. Use when: truncation or exact zeros remove support; exchanging arguments changes the requirement.

Expand the candidate set

New task: permit $q = p$ as another candidate. Every log ratio is now zero for that candidate. Check: both objectives attain zero at $p$ , beating the original choices. Use when: a richer approximation family becomes available; reverse KL does not inevitably select one mode.

Failure and transfer

For $p = (0.9, 0.1)$ and $q = (0.5, 0.5)$ , the wrong average $E_{q} [lo g (p / q)]$ is $- 0.510826$ . The actual $KL (p ∥ q)$ is $0.368064$ . Wrong sampling weights changed the quantity, not the nonnegativity theorem.

Reconstruct: fixed $p$ has entropy $0.7$ nats and cross-entropy $1.2$ nats. What is KL?

Answer

It is $0.5$ nats. Minimizing cross-entropy over $q$ also minimizes forward KL only because this $p$ , and hence its entropy, stays fixed.

Why the divergence is nonnegative

For positive $r$ , $- lo g r \geq 1 - r$ . Apply this with $r = q (x) / p (x)$ wherever $p (x) > 0$ :

KL (p ∥ q) \geq p (x) > 0 \sum (p (x) - q (x)) = 1 - p (x) > 0 \sum q (x) \geq 0.

Equality requires $q (x) = p (x)$ on the positive support and no remaining candidate mass outside it. Zero candidate probability at positive target mass instead gives infinite KL immediately. The coding interpretation compares expected ideal code lengths under the same sampling law; base-two logarithms give bits rather than nats.

Continuous laws and empirical samples

For general measures, finite KL requires absolute continuity $p ≪ q$ and suitable integrability. For densities, use $\int p lo g (p / q)$ against the common base measure. An atomic empirical measure has infinite KL to a continuous law assigning zero probability to every observed point. Empirical negative log density remains meaningful; it is not that literal finite empirical-to-continuous KL identity.

Common usage in ML

Use case	Direction
Classification cross-entropy loss	Forward $KL (data ∥ model)$
Variational inference	Approximate posterior $q (z ∣ x)$ versus model posterior $p (z ∣ x)$
Policy regularization	Name both policies, expectation, and penalty direction explicitly
Knowledge distillation	Teacher versus student distributions at the specified temperature
t-SNE	High-dimensional versus low-dimensional pairwise probability distributions

PPO’s clipped surrogate is not generally an exact KL constraint. Information bottleneck objectives also use distinct joint, marginal, and conditional laws; name them before applying an identity.

Common pitfalls

Jensen-Shannon divergence averages $KL (p ∥ m)$ and $KL (q ∥ m)$ with $m = (p + q) /2$ . That symmetric bounded quantity is not KL with swapped arguments or a universal GAN objective. Finite-sample density estimates introduce their own support choices and estimation bias. Changing the feasible family can matter as much as divergence direction.