Backpropagation

Summary

Backpropagation accumulates a scalar loss’s sensitivity through every recorded route into each parameter. It applies reverse-mode automatic differentiation to a computation graph, reusing forward values needed by local derivative rules. For standard operations with comparable derivative costs, one scalar-output reverse pass costs $O (F)$ for forward work $F$ . It computes gradients, not a parameter update or a guarantee of optimization progress.

The algorithm

Execute operations in dependency order and retain the values their backward rules need. Initialize all accumulated sensitivities to zero and seed $\overline{L} = 1$ . Visit operations in reverse dependency order, adding each contribution rather than overwriting earlier contributions:

\overline{z} + = J_{f} (z)^{⊤} \overline{y}, y = f (z) .

The matrix-calculus pullback explains this transposed-Jacobian rule. Local reverse products usually avoid constructing the full Jacobian.

For $x = 2$ , $w = 3$ , let $z_{1} = x w = 6$ , $z_{2} = z_{1} + 1 = 7$ , and $L = z_{2}^{2} = 49$ . The reverse chain gives $\overline{z}_{2} = 14$ , $\overline{z}_{1} = 14$ , $\overline{x} = 14 w = 42$ , and $\overline{w} = 14 x = 28$ . These are the exact values and arrows in the diagram.

Spatial intuition

The same upstream gradient meets different cached multipliers

Derivation

Read it this way: each reverse arrow multiplies one upstream gradient by one local derivative. The graph is traversed once in each direction; no full Jacobian is built.

Check: Why does w receive gradient 28 while x receives 42?

Read it this way: A cached forward value determines each local derivative. Multiplying by the shared upstream signal gives different parameter gradients.
If x doubles while the upstream signal stays fixed, which parameter gradient doubles?

Vector-Jacobian products (VJPs)

For $y = f (z)$ , an incoming output gradient $g_{y}$ produces $g_{z} = J_{f}^{⊤} g_{y}$ . Primitive rules combine to implement this vector-Jacobian product without materializing a network-wide Jacobian.

Reuse an Intermediate

New task: Differentiate $t = x w$ and $L = t + t$ .

Picture and change: Two outgoing routes contribute to the same intermediate; sum them before propagating through multiplication.

Check: $\overline{t} = 1 + 1 = 2$ , hence at $x = 2, w = 3$ , gradients are $\overline{x} = 6$ , $\overline{w} = 4$ . Overwriting one contribution would incorrectly halve both answers.

Use when: Parameters or intermediate results feed multiple operations. Repeated references are multiple derivative contributions even when the graph node is stored once.

Memory cost

New task: Compute the same gradient with less retained activation memory.

Picture and change: Save selected boundaries and recompute needed forward values during backward.

Check: Recomputing $z_{1} = 6$ and $z_{2} = 7$ in the original example still gives gradients $42, 28$ .

Use when: Activation checkpointing can trade extra work for a lower memory peak. Recomputation must reproduce required random choices and state-dependent behavior. Storing lower-precision activations is a separate numerical trade-off, not exact checkpointing. Total memory depends on retained tensors, lifetimes, parameters, optimizer state, and temporary workspace, not depth alone.

Connection to reverse-mode autodiff

For a full Jacobian with $P$ inputs and $M$ outputs, forward mode needs roughly $P$ directional sweeps. Reverse mode needs roughly $M$ output-seeded sweeps. One scalar loss therefore favors reverse mode for many parameters; a few requested directional derivatives may favor forward mode.

What backprop does NOT do

It does not choose a learning rate, regularizer, or optimizer.
It is not specific to neural networks; differentiable programs can use the same accumulation rule.
It does not differentiate through unrecorded operations or automatically supply a derivative for discrete choices.
It does not prove convergence; even an exact gradient can be used in a non-descending finite update.

Common pitfalls

Unintended accumulation. Framework parameter gradients may persist across backward calls; reset them at the intended optimization boundary.
Saved-value reuse. Backward may release needed saved tensors; a second backward through the same graph can require explicit retention.
Detached routes. detach() stops gradient flow through that route.
Retained graphs. Holding differentiable loss objects can retain graph metadata or saved values; log detached scalars when appropriate.
Leaf confusion. Requiring gradients and being a leaf are different properties; retained intermediate gradients need their own policy.

Reconstruct

Why must a shared node wait until every downstream contribution has arrived?

Its sensitivity is the sum over downstream routes. Propagating a partial sum and then discarding the node loses later routes; reverse dependency order prevents that mistake.

Matrix calculus. The chain rule formalism.
Activation checkpointing. Memory optimization for backprop.