Normalizing flows

Summary

A normalizing flow transports a base density through invertible transformations, accounting for their local volume changes. For the differentiable bijections here, density evaluation uses the inverse coordinate and the inverse Jacobian’s absolute determinant:

lo g p_{X} (x) = lo g p_{Z} (f^{- 1} (x)) + lo g det \frac{\partial f ^{- 1}}{\partial x} .

An invertible coordinate is not a nontrivial VAE-style posterior distribution. Tractable discrete-layer likelihood and fast sampling depend on the selected architecture; neither property uniquely belongs to flows.

The change-of-variables formula

Require a differentiable inverse and nonzero Jacobian determinant on the relevant support. For $z = f^{- 1} (x)$ :

p_{X} (x) = p_{Z} (z) \cdot det J_{f^{- 1}} (x) = p_{Z} (z) \cdot ∣ det J_{f} (z) ∣^{- 1} .

For a composition of $K$ flows, the log-determinant decomposes additively:

lo g p_{X} (x) = lo g p_{Z} (z) - k = 1 \sum K lo g ∣ det J_{f_{k}} (z_{k - 1}) ∣ .

Evaluate each layer’s determinant at its actual intermediate state. Evaluate standard Gaussian density at $z = (1, 2)$ . Set $f_{1} (z) = (2 z_{1}, z_{2})$ and $f_{2} (u) = (u_{1}, 1.5 u_{2})$ . The states are $(1, 2) \to (2, 2) \to (2, 3)$ , with determinant factors two and $1.5$ . The local area triples while probability mass is conserved, so $p_{X} (2, 3) = p_{Z} (1, 2) /3$ . Gaussian log density gives:

lo g p_{X} (2, 3) = - lo g (2 π) - 2.5 - lo g 3 \approx - 5.43649.

Change of variables

Expanding volume lowers density while preserving probability mass

Derivation

Read it this way: forward transforms stretch a local patch without changing its probability mass. Inverse density evaluation divides by the total volume multiplier. Here area multiplies by three and log density loses log three. Dots illustrate mass; they are not point-probability density values. Original construction follows Real NVP and the flows review.

Check: Why does tripling occupied volume divide density by three while leaving total probability mass unchanged?

Read it this way: The same four dots carry mass 1 before and after expansion. Doubling the drawn area changes density from 1 to 0.5, not the total probability mass.
What happens to density if the same mass occupies twice the area?

Inverting $(2, 3)$ returns $(1, 2)$ ; multiplying its transformed density by tripled local area recovers the same mass.

RealNVP / coupling layers (the workhorse)

Split $x = (x_{a}, x_{b})$ . Then:

y_{a} = x_{a}, y_{b} = x_{b} ⊙ exp (s (x_{a})) + t (x_{a})

The Jacobian is block triangular, with log absolute determinant $\sum_{i} s (x_{a})_{i}$ . Inverse reconstruction uses $x_{a} = y_{a}$ and $x_{b} = (y_{b} - t (y_{a})) ⊙ exp (- s (y_{a}))$ . The internal s/t networks need not themselves be invertible.

Stack many coupling layers, alternating which half passes through, with shuffles or 1×1 convs between them.

Change a Scale into Coupling

New task: Use $x_{1} = z_{1}$ , $x_{2} = 2 z_{2} + 1$ through constant $s = lo g 2$ , $t = 1$ .

Picture and change: Apply the triangular coupling and evaluate its explicit inverse.

Check: $(1, 2)$ maps to $(1, 5)$ , determinant two; inverse second coordinate is $(5 - 1) /2 = 2$ .

Use when: A structured invertible layer needs a tractable determinant without inverting its internal neural networks.

Change the Required Fast Direction

New task: Prioritize density scoring in three coordinates over generation latency.

Picture and change: In MAF, known x prefixes support masked parallel inverse-coordinate evaluation. Sampling must resolve the x prefixes sequentially; IAF reverses this directional convenience.

Check: Generating x1, x2, x3 takes three dependent coordinate stages in this ordering.

Use when: Density queries and sample generation have different serving priorities.

The map $x = z^{2}$ is not a bijection on the real line: positive and negative z collide. Using only one inverse branch in the stated formula loses probability mass.

Common flow families

Family	Structure	Important limit
Coupling	Triangular Jacobian; NICE is an additive special case	Expressiveness needs suitable compositions
MAF/IAF	Autoregressive coordinate dependence	Fast direction differs
Continuous time	ODE and divergence integration	Solver/trace estimation introduces numerical or Monte Carlo error
Invertible channel mixing	Learned nonsingular linear transform	Retain determinant and conditioning checks

When to use flows in 2026

Compare density estimation, simulation-based conditional inference, forecasting, and generative tasks under their actual accuracy and compute criteria. A high likelihood does not automatically identify in-distribution semantics or provide a reliable anomaly/OOD score. Do not infer a universal image-quality or training-speed ranking from model-family names.

Common pitfalls

Wrong determinant direction: density evaluation adds inverse log determinant, equivalently subtracting forward log determinant.
Discrete observations: dequantization models a continuous auxiliary distribution; relate its density objective to discrete likelihood explicitly.
Numerics: finite precision, nearly singular transforms, and ODE tolerances can invalidate an unqualified “exact” claim.

Variational autoencoders. Alternative latent-variable generative model.
Autoregressive vs. diffusion. Broader paradigm comparison.