Residual connections

Summary

A residual block adds a learned change to a shortcut: $y = x + f (x)$ . The identity branch supplies a direct gradient contribution, not a guarantee about the combined gradient.

Take $x = 2$ , $f (x) = 0.1 x$ and $L = y^{2} /2$ . Find the output and the two contributions to $\partial L / \partial x$ .

The mechanism

A residual block:

y = x + f(x)

The input goes to both branches. Addition joins their values in the forward pass and splits the upstream gradient in backward.

One input, two contributions

Addition copies the upstream gradient; each branch then applies its own derivative

Derivation

Read it this way: the forward branches add to 2.2. Backward copies the upstream 2.2 to both branches. The residual branch scales it by 0.1; the shortcut leaves it unchanged. Their contributions add to 2.42.

Check: Why does the add node copy 2.2 into both branches? Why does the residual branch return only 0.22?

Read it this way: With upstream 2.2, the identity branch returns 2.2. A residual branch with derivative 0.1 returns 0.22, so the input receives 2.42.
Does the identity path receive only half of the upstream gradient?

References: Deep Residual Learning and Identity Mappings in Deep Residual Networks.

The loss is $2. 2^{2} /2 = 2.42$ , giving upstream gradient $g_{y} = 2.2$ . The shortcut contributes $2.2$ ; the residual branch contributes $0.22$ . Directly differentiating $L = (1.1 x)^{2} /2$ gives $1.21 x = 2.42$ at $x = 2$ too.

For vector inputs, the diagram’s row-gradient convention gives

g_{x} = g_{y} (I + J_{f}) .

One contribution crosses no residual operation. The sum still depends on $J_{f}$ .

Change the shortcut

1. Change output dimension

New task: map $x = (1, 2)^{T}$ into three channels with $f = 0$ .

Picture and change: replace the identity wire by $P = 101011$ , giving $y = P x + f (x)$ . The branches must produce matching shapes before addition.

Check: $y = (1, 2, 3)^{T}$ . Row upstream gradient $(1, 1, 1)$ returns $g_{y} P = (2, 2)$ , not the unchanged upstream vector.

Use when: channels or spatial resolution change across a block.

2. Move normalization outside the addition

New task: compare pre-norm $x + f (LN (x))$ and post-norm $LN (x + f (x))$ , with $f = 0$ . LayerNorm has unit gain and zero offset.

Picture and change: add a common shift $c (1, 1)$ to $x$ . Pre-norm retains it through the shortcut; post-norm removes it by centering.

Check: the directional derivative along $(1, 1)$ is $(1, 1)$ for pre-norm and zero for post-norm.

Use when: reasoning about gradient paths across normalization boundaries.

The two contributions can cancel

Set $f (x) = - x$ . Then $y = 0$ and $I + J_{f} = 0$ . The total input gradient is zero even though the identity contribution exists. An identity branch does not prove a nonvanishing total gradient.

Try repeating the block

Does repeatedly applying $f (x) = 0.1 x$ prevent gradient explosion?

Check your answer

No. Each block has local gain $1.1$ . Across $N$ identical scalar blocks, the derivative is $1. 1^{N}$ .

Why it works (intuitions)

The ResNet paper studied degradation in deeper plain networks, including worse training error. An additive block can represent identity by setting its residual function to zero. Veit et al. analyze paths of differing depths; this is not an ensemble of independently trained models.

Pre-norm vs. post-norm in transformers

Their Jacobians are $I + J_{f} J_{LN}$ for pre-norm and $J_{LN} (I + J_{f})$ for post-norm. Matrix order matters. Neither expression guarantees bounded gradients through a whole network. See Xiong et al. for normalization placement under stated training and initialization assumptions.

Bottleneck blocks

A bottleneck reduces channels before the expensive spatial convolution, then expands them:

y = x + Conv1x1 ↓ → Conv3x3 → Conv1x1 ↑ (x)

Ignoring biases, its convolution parameters total $C_{in} C_{m} + 9 C_{m}^{2} + C_{m} C_{out}$ . Savings depend on widths, spatial sizes and the comparison block, not a fixed factor or accuracy guarantee.

Where residuals show up

Architecture	Where
ResNet, ResNeXt, Wide ResNet	Every block
U-Net	Long skips may concatenate; distinguish those from additive residual blocks
Transformer (encoder & decoder)	Around attention and FFN sub-blocks
Diffusion U-Nets	Both within blocks and across encoder-decoder
MLP-Mixer, ConvNeXt	Block residuals

Common pitfalls

Some deep-network recipes scale residual contributions with depth; the scale must match their initialization and training assumptions. Addition, saved activations and projection shortcuts have memory or compute costs.