Weight decay vs. L2 regularization

Summary

L2 regularization adds a penalty $\frac{\lambda }{2}\Vert \theta {\Vert }^2$ to the loss; weight decay multiplies parameters by $(1-\eta \lambda )$ at each step. Under vanilla SGD they are mathematically equivalent. Under adaptive optimizers like Adam they are not. And the difference is large enough that AdamW (Loshchilov & Hutter, 2019) is now the default for transformer training.

The two formulations

L2 (penalty added to loss)

$$L_{\text{total}}(\theta )=L_{\text{data}}(\theta )+\frac{\lambda }{2}\Vert \theta {\Vert }^2$$

Gradient: $\nabla L_{\text{total}}=\nabla L_{\text{data}}+\lambda \theta$. Update under SGD: ${\theta }_{t+1}={\theta }_t-\eta (\nabla L_{\text{data}}+\lambda {\theta }_t)$ $=(1-\eta \lambda ){\theta }_t-\eta \nabla L_{\text{data}}$.

Weight decay (multiplicative shrink)

$${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta \nabla L_{\text{data}}$$

Same expression. Under vanilla SGD, the two are identical.

Why they diverge under Adam

Adam scales the gradient per parameter using its moment history. With L2, the penalty is added before those moments are updated:

$$g_t=\nabla L_{\text{data}}({\theta }_t)+\lambda {\theta }_t,{\theta }_{t+1}={\theta }_t-\eta \cdot \frac{{\hat{m}}_t(g_{1:t})}{\sqrt{{\hat{v}}_t(g_{1:t})}+\epsilon }.$$

The penalty therefore enters both moment histories and inherits Adam’s coordinate-wise scaling. In the fixed-preconditioner view, its contribution is proportional to $\lambda {\theta }_t/\sqrt{{\hat{v}}_t}$: larger for coordinates with a small denominator and smaller for coordinates with a large denominator. Regularization is now coupled to gradient history.

Learning objective

Adam with L2 sends shrinkage through the adaptive denominator; AdamW routes it around.

Follow λθ: through Adam's denominator in L2, around it in AdamW.

AdamW decouples them: apply Adam to the data loss only, and then shrink the parameters multiplicatively as a separate step:

$${\theta }_{t+1}=(1-\eta \lambda ){\theta }_t-\eta \cdot \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t+\epsilon }}$$

The shrink term has no $1/\sqrt{v_t}$ scaling. This recovers the SGD-equivalent behavior.

Empirical impact

Loshchilov & Hutter (2019) and many follow-up benchmarks show AdamW generalizes meaningfully better than Adam-with-L2 across vision and NLP. The exact gain depends on the task; on transformer LLM training the gap is large enough that essentially all modern training uses AdamW.

What to skip

Common practice: do not decay biases, LayerNorm parameters, or embeddings. These are 1D parameters with different statistical roles, and decaying them often hurts. Standard implementations construct two parameter groups: {decay: linear weights, conv kernels} and {no decay: biases, norms, embeddings}.

Common pitfalls

• Using Adam with weight_decay > 0 in PyTorch. This applies L2-as-gradient, not AdamW. Use AdamW explicitly.
• Decaying bias and LayerNorm parameters. Hurts performance; exclude them via parameter groups.
• Picking $\lambda$ from a CNN recipe. $\lambda =1\text{e-}4$ for ResNets; $\lambda =0.1$ for AdamW transformer pretraining (with the no-decay carve-out). Different scale, different rule of thumb.
• Forgetting that decay scales with LR. Effective shrink per step is $\eta \lambda$. Halving LR halves effective decay; you may need to compensate.

Related

• Adam and AdamW. For the optimizer-side derivation.
• Regularization. Broader survey of regularization techniques.