L2 and weight decay: follow the shrinkage path

Summary

L2 adds a penalty to the objective. Decoupled weight decay shrinks parameters outside the optimizer’s gradient history. Their updates coincide for vanilla SGD, but that algebra does not automatically extend to momentum or Adam.

Task: start two parameters at $[1, 1]$ , with data gradient $[0.2, 0.2]$ , learning rate $η = 0.01$ , and $λ = 0.1$ . Freeze adaptive denominators at $[1, 10]$ to isolate one step. Does routing $λ θ$ through those denominators give the same result as direct decay?

The two formulations

L2 (penalty added to loss)

L_{total} (θ) = L_{data} (θ) + \frac{λ}{2} ∥ θ ∥^{2}

Writing $g = \nabla L_{data} (θ)$ , the penalized gradient is $g + λ θ$ . Vanilla SGD gives the exact expansion

θ - η (g + λ θ) = (1 - η λ) θ - η g .

Weight decay (multiplicative shrink)

Direct decay uses the expression on the right. Equality requires the same learning rate, penalty convention, parameter groups, and old-state evaluation, with no momentum or adaptive scaling.

Why they diverge under Adam

Coupled L2 feeds $g + λ θ$ into both Adam moment buffers. Decoupled AdamW feeds only $g$ into them. The history itself changes, not merely a final division.

For the frozen-denominator fixture, isolate the paths without updating moments:

Quantity	Coordinate 1	Coordinate 2
Coupled data-plus-penalty step $η (g + λ θ) / s$	$0.003$	$0.0003$
Coupled next parameter	$0.997$	$0.9997$
Direct decay $η λ θ$	$0.001$	$0.001$
Decoupled data step $η g / s$	$0.002$	$0.0002$
Decoupled next parameter	$0.997$	$0.9988$

The equal weights receive equal fractional direct decay despite unequal adaptive denominators. This illustration is not a complete coupled-Adam trajectory; real coupled penalties also change both moments.

Learning objective

Adam rescales coupled L2 through its moments; AdamW applies decay directly

Derivation

Follow λθ: through Adam's denominator in L2, around it in AdamW.

follow λθ. In Adam with L2, it joins the gradient before the adaptive machinery and changes the moment histories. AdamW's upper bypass instead applies direct fractional decay. Numerical denominator comparisons freeze the preconditioner; they are not full coupled-Adam trajectories. Original schematic checked against Loshchilov and Hutter (2019) and the PyTorch AdamW algorithm.

Check: Why does Adam scale equal L2 penalties differently, while AdamW applies equal direct decay fractions?

Read it this way: For equal weights, coupled L2 passes through different fixed preconditioners. The illustrative decoupled decay remains 0.001 for either history.
Which route changes if the preconditioner grows from one to ten?

For AdamW, compute corrected moments from the data gradients and evaluate both contributions against the old parameter:

θ_{t + 1} = (1 - η_{t} λ) θ_{t} - η_{t} \frac{m _{t}}{v _{t} + ϵ} .

Epsilon is outside the square root in this convention. Shrinking an already updated parameter instead introduces a cross-term; it is not the displayed update.

Remove adaptive scaling

New task: replace denominators $[1, 10]$ with $[1, 1]$ , without momentum. Picture and change: both paths reduce to the vanilla-SGD expansion. Check: both produce $[0.997, 0.997]$ . Use when: the actual optimizer satisfies those assumptions, not merely because its name contains SGD.

Exempt one parameter

New task: exclude coordinate two from direct decay while retaining its data gradient. Picture and change: set only its shrink contribution to zero. Check: the decoupled result becomes $[0.997, 0.9998]$ , instead of $[0.997, 0.9988]$ . Use when: the architecture explicitly exempts that parameter role; tied tensors must not receive duplicate updates.

Failure and transfer

With momentum, even zero current data gradient can leave a nonzero history-driven update. The vanilla proof cannot identify that movement with direct shrinkage. With zero adaptive updates, cumulative decay is $\prod_{t} (1 - η_{t} λ)$ , so the schedule matters.

Reconstruct: what happens when $η$ halves from $0.01$ at $θ = 1$ , $λ = 0.1$ ?

Answer

One direct-decay step changes from $0.999$ to $0.9995$ . The decay exposure halves for that step; this says nothing about stored-gradient movement.

Empirical impact

Loshchilov and Hutter distinguish decoupled decay from adaptive L2 and report experiments. Their results are not a guarantee of improved generalization on every architecture or training regime. Compare tuned coefficients and schedules under a matched evaluation contract.

What to skip

Review parameter roles rather than applying a universal no-decay list. Biases and normalization scales are often treated separately; embedding tables are usually matrices, not one-dimensional parameters. An embedding can also be tied to an output projection, so membership must follow tensor identity. Assign every trainable parameter exactly once and retain the architecture’s chosen exclusions.

Common pitfalls

Inspect the installed API’s decay option; some Adam implementations expose a decoupled mode.
Loss reduction conventions change the relative scale of an objective penalty.
A copied coefficient does not preserve exposure when step count or learning-rate schedule changes.
The term is ordinary fractional shrinkage when $0 \leq η_{t} λ < 1$ ; larger products can erase or flip a parameter.

Adam and AdamW. For the optimizer-side derivation.
Regularization. Broader survey of regularization techniques.