Skip to content
mlmentorship

Explain backprop in your own words

Backprop is reverse-mode automatic differentiation. It reuses forward-pass values to compute all parameter gradients at roughly one additional forward-pass cost.

Published · 7 min read ·Core ·Mixed

ML breadth · active recall

Practice before you read

8 minutes. Explain the mechanism, why it works, when it fails, and one alternative.

Back to my workbook How practice works

ML breadth · closed-book attempt

Explain backprop in your own words

Explain the mechanism, why it works, when it fails, and one alternative.

Problem

Context. A differentiable computation produces a scalar loss from parameters and intermediate values that may feed several later operations. For a small illustration, take loss = x * x + x at x = 2. Task. Trace the forward values and reverse derivatives, explain gradient accumulation, and relate the procedure to reverse-mode automatic differentiation. Discuss computational cost, saved intermediates, and common ways gradient flow is interrupted. Constraints. Distinguish the derivative calculation from the optimizer update. Account for every path to a reused value, and state conventions for nondifferentiable operations rather than assuming all graphs are smooth.

08:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: ML breadth at every level.

Explain backpropagation as a computation over a graph, including how gradients from multiple uses of a value combine.

Problem

Context. A differentiable computation produces a scalar loss from parameters and intermediate values that may feed several later operations. For a small illustration, take loss = x * x + x at x = 2.

Task. Trace the forward values and reverse derivatives, explain gradient accumulation, and relate the procedure to reverse-mode automatic differentiation. Discuss computational cost, saved intermediates, and common ways gradient flow is interrupted.

Constraints. Distinguish the derivative calculation from the optimizer update. Account for every path to a reused value, and state conventions for nondifferentiable operations rather than assuming all graphs are smooth.

What an L4 answer sounds like

“Backprop computes gradients of the loss with respect to every weight using the chain rule. Starting from the output, you propagate the error backwards through each layer.”

Correct, but generic. You’ve memorized the textbook line, not internalized the algorithm.

What an L5 answer sounds like

“Backprop is reverse-mode automatic differentiation applied to the computation graph of a neural network.

The forward pass computes the loss while caching intermediate activations. The backward pass starts at the loss with gradient 1, then walks backward through the cached graph. At each operation, it multiplies the upstream gradient by the local Jacobian of that op with respect to its inputs (the chain rule), and accumulates gradients at parameters.

The key efficiency insight: reverse-mode evaluates one forward and one backward pass per output. A naive ‘compute gradient by changing each parameter’ approach would cost O(P) forward passes for P parameters. Reverse-mode is O(1) forward + O(1) backward to get all P gradients.”

Learning objective

One reverse sweep computes all parameter gradients for this scalar loss

One scalar loss sends contributions to every ancestorThis illustrative graph uses u=a+3b and loss 2u. One backward flow gives shared sensitivity two, then parameter gradients two and six. Shared contributions accumulate before propagation.One loss sends one accumulatedreverse flowLossdL/du = 2uagradient 2bgradient 6
Derivation
One forward pass and one reverse sweep through a three-operation graph A solid path carries input x through operations f1, f2, and f3 to scalar loss L. Parameters theta 1, theta 2, and theta 3 each enter one operation. Activations a1 and a2 are explicitly saved during the forward pass. A dashed reverse path starts at dL over dL equals 1, moves left through all three operations using the saved activations, and branches to produce dL over d theta 3, dL over d theta 2, and dL over d theta 1 in the same sweep. FORWARD · COMPUTE ONCE AND SAVE WHAT LOCAL DERIVATIVES NEED x f₁uses θ₁ a₁saved f₂uses θ₂ a₂saved f₃uses θ₃ L BACKWARD · ONE REVERSE SWEEP, NOT ONE SWEEP PER PARAMETER seed ∂L/∂L = 1 VJP uses saved a₂ VJP uses saved a₁ ∂L/∂θ₃accumulate at parameter ∂L/∂θ₂accumulate at parameter ∂L/∂θ₁accumulate at parameter
follow the solid arrows once to compute the scalar loss and save a₁ and a₂. Then start at ∂L/∂L = 1 and follow the dashed arrows left: each local vector-Jacobian product reuses a saved value and emits that operation's parameter gradient. The reverse path is traversed once, not once per θ. Original schematic checked against the automatic-differentiation survey and PyTorch autograd mechanics.
Check: Why does adding more parameters inside these operations not require another full reverse sweep per parameter?
Read it this way: This illustrative graph uses u=a+3b and loss 2u. One backward flow gives shared sensitivity two, then parameter gradients two and six. Shared contributions accumulate before propagation.
Does each parameter require a separate complete reverse traversal?

This is L5. You’ve explained it as auto-differentiation, mentioned the activation caching, and quantified the efficiency win.

What an L6 answer sounds like

“…a few practical things worth adding:

Memory is the cost. The backward pass needs the cached activations from the forward pass. For deep networks with large activations, this dominates GPU memory. Activation checkpointing trades compute for memory by recomputing activations during backward instead of storing them.

Backprop is exact, not approximate. Unlike numerical differentiation, it has no truncation error. The only error is floating-point.

It composes through any differentiable op. This is why frameworks (PyTorch, JAX) implement autograd as a graph of primitive ops with known local Jacobians. Custom ops just need to define forward and the local Jacobian (vector-Jacobian product); the framework composes the rest.

What breaks it in practice: non-differentiable ops (argmax, hard threshold), vanishing gradients in deep networks (ReLU, residuals, normalization mitigate), exploding gradients (gradient clipping mitigates), and detached tensors silently breaking gradient flow.”

Tells that get you a strong-hire vote

  • You frame it as reverse-mode auto-diff, not just “the chain rule.”
  • You mention activation caching and the memory cost.
  • You connect to activation checkpointing as the standard memory-compute trade.
  • You distinguish backprop (exact) from numerical differentiation.
  • You mention detached tensors as a common bug source.

Tells that get you down-leveled

  • “It’s the chain rule” with no further detail.
  • Confusion about which gradients flow where (e.g., gradients flow through cached activations, not just weights).
  • No awareness of the memory cost.
  • Calling it “back-propagation of errors” without explaining what’s actually being computed.

Common follow-up

“Why is reverse-mode preferred over forward-mode auto-diff for neural networks?”

The L6 answer:

“Forward-mode computes the derivative of one input with respect to all outputs in one pass. Reverse-mode computes the derivative of one output (the loss) with respect to all inputs (parameters) in one pass. Neural network training has many parameters and one scalar loss, so reverse-mode is the right shape. For small models with many outputs and few inputs, forward-mode can be faster.”

Related: Adam and AdamW and debug a model that is not learning.

Found an error or stale claim? Report a correction or send a private note.