Variational autoencoders: calculate the bound and its gap

Summary

A variational autoencoder fits a latent-variable model using approximate posterior inference. Its encoder estimates latent uncertainty for observed data; its decoder specifies an observation distribution, not merely a reconstructed mean.

Task: observe $x = 2$ under $p (z) = N (0, 1)$ and $p (x ∣ z) = N (z, 1)$ . Use approximate posterior $q (z ∣ x) = N (1, 1/4)$ , where the second argument denotes variance. Compute the training bound and compare it with the exact log-likelihood. This scalar teaching model is solvable; neural models often make the marginal integral difficult.

The model

Generally, $p_{θ} (x, z) = p (z) p_{θ} (x ∣ z)$ and the encoder is $q_{ϕ} (z ∣ x)$ . Parameters $θ$ define the generative model; $ϕ$ defines its approximate inference network. One encoder shares inference parameters across observations: amortization does not make each inferred posterior exact.

Read the training lane and generation lane as different computations.

One model, two execution paths

Generation samples the prior without an observed input or encoder

Derivation

observed x enters the encoder on the solid training path. A posterior sample reaches the decoder; reconstruction scores the conditional density of that same x. The KL term compares the approximate posterior with the prior. The dashed generation path instead samples the prior and uses the decoder without an observed x. Sources: Kingma and Welling, Rezende et al., and the VAE tutorial.

Check: Why does generation need neither an observed x nor q(z|x), unlike training?

Read it this way: Training infers z from observed x through the encoder. Generation instead samples z from the prior and decodes it without q(z|x).
Which latent source is used when no observed input exists?

The ELBO

For these everywhere-positive Gaussian densities, insert $q$ and apply Jensen’s inequality:

lo g p_{θ} (x) = lo g E_{q} [\frac{p _{θ} ( x , z )}{q ( z ∣ x )}] \geq E_{q} [lo g p_{θ} (x, z) - lo g q (z ∣ x)] = E_{q} [lo g p_{θ} (x ∣ z)] - KL (q ∥ p (z)) .

Call the final expression $L$ , the evidence lower bound, or ELBO. Bayes’ rule also gives the exact gap identity:

lo g p_{θ} (x) = L + KL (q_{ϕ} (z ∣ x) ∥ p_{θ} (z ∣ x)) .

The penalty inside $L$ compares $q$ with the prior; the gap compares $q$ with the model posterior. These are different distributions.

For the fixture, $E_{q} [(2 - z)^{2}] = (2 - 1)^{2} + 1/4 = 1.25$ . For a Gaussian encoder with mean $μ$ and variance $s^{2}$ , the prior KL is $(μ^{2} + s^{2} - 1 - lo g s^{2}) /2$ .

Quantity, natural-log units	Calculation	Value
Expected reconstruction log density	$- \frac{1}{2} lo g (2 π) - 0.625$	$- 1.54394$
Prior KL	$(1 + 1/4 - 1 - lo g (1/4)) /2$	$0.81815$
ELBO	Reconstruction minus prior KL	$- 2.36209$
Exact log-likelihood, $x \sim N (0, 2)$	$- \frac{1}{2} lo g (4 π) - 1$	$- 2.26551$

The exact posterior is $N (1, 1/2)$ . Directly computing its KL from $q$ gives $(1/2 - 1 - lo g (1/2)) /2 = 0.09657$ , exactly the likelihood-minus-bound gap.

Improve inference variance

New task: change $q$ ‘s variance from $1/4$ to the exact posterior variance $1/2$ . Update both reconstruction and prior KL, not just the penalty. Check: reconstruction becomes $- 1.66894$ , prior KL $0.59657$ , and ELBO $- 2.26551$ . The posterior gap is zero. Use when: the inference family can better represent the model posterior.

Change the KL weight

New task: keep the original $q$ but optimize reconstruction minus $β$ times prior KL, with $β = 1/2$ . Check: the objective becomes $- 1.95301$ , exceeding the exact log-likelihood $- 2.26551$ . It is not the ordinary ELBO; its usual lower-bound guarantee cannot be asserted. Use when: deliberately changing a representation or rate-distortion tradeoff rather than claiming the same likelihood objective.

The reparameterization trick

Write $z = μ_{ϕ} (x) + σ_{ϕ} (x) ⊙ ϵ$ , with parameter-independent $ϵ \sim N (0, I)$ . For log variance $ℓ$ , use $σ = exp (ℓ /2)$ , not $exp ℓ$ . The randomness remains; its sampling law no longer depends on $ϕ$ . See the pathwise-gradient derivation for conditions and alternatives.

Reconstruct: which observed $x$ enters the encoder during unconditional generation?

Answer

None. Sample $z$ from the prior, then sample or summarize the decoder distribution $p_{θ} (x ∣ z)$ . Reconstruction instead conditions on a supplied observation through the encoder.

What VAEs are good and bad at

Continuous latent models can support interpolation and learned compression, including latent representations used by downstream diffusion models. Their usefulness depends on the learned distribution and reconstruction contract, not continuity alone. Gaussian squared error corresponds to a specified decoder variance; it is not a universal reconstruction likelihood. Blur depends on that likelihood, uncertainty, architecture, and optimization, not simply the VAE name. No universal stability or sample-quality ranking follows against GANs or diffusion.

Common pitfalls

Posterior collapse means inferred latents carry little observation-specific information and the decoder can ignore them. An expressive autoregressive decoder can still produce varied outputs; collapse does not imply a constant image. Annealing, free bits, decoder changes, and inference improvements alter different parts of training and offer no universal cure. Not using a pathwise estimator does not uniquely force REINFORCE; other estimators depend on the latent family. Check likelihood sums, minibatch reductions, and KL units before interpreting their relative weight.

Variants

Beta-VAE: changes the KL weight; disentanglement is not guaranteed by that coefficient alone.
VQ-VAE: introduces discrete codebooks and a different training construction, not Gaussian reparameterization unchanged.
IWAE: uses an importance-weighted objective with its own support and sampling conditions.
NVAE and hierarchical VAEs: introduce multiple conditional latent levels, retaining a distinct factorization.

The IWAE bound is an expectation, not an exact likelihood correction. A single Monte Carlo realization need not lie below log-likelihood.

Reparameterization trick. The gradient enabler.
Autoregressive vs. diffusion. Alternative generative paradigms.