Why does Adam sometimes generalize worse than SGD?

Compare Adam and SGD through controlled optimization and generalization experiments rather than assuming one optimizer universally wins.

Problem

Context. A workload appears to generalize differently under two optimizers; the coordinate-change fixture below probes one proposed sharpness explanation.

Task. Design a fair comparison, separate observed performance from its explanation, and analyze whether reparameterization changes the predictor’s outputs. Identify evidence that would support or reject a proposed mechanism.

Constraints. Make data splits, initialization, tuning budgets, schedules, and regularization conventions explicit. Parameter-space curvature depends on coordinates; do not treat it alone as proof of different function-space behavior or generalization.

A toy predictor outputs $w$ , with loss $L (w) = (w - 1)^{2} /2$ . Reparameterize it as $w = 10 ϕ$ . Compare the equivalent minima $w = 1$ and $ϕ = 0.1$ . Does a sharper parameter-space curve imply different predictions?

The sharpness caveat

The basin gets sharper without changing the model's predictions

Derivation

Read it this way: coordinate curvature changes from one to one hundred. Both minima still predict one. These exact toy curves show coordinate dependence, not measured optimizer performance.

Check: Did predictions change when the plotted basin became sharper?

Read it this way: These two parameter coordinates represent the same scalar prediction error. Curvature alone is not invariant to reparameterization.
Does relabeling a=2b alter the represented scalar prediction?

Reparameterization caveat: Dinh et al. (2017). This scalar construction is calculated independently.

Calculate the curvature

In $w$ coordinates, $L^{''} (w) = 1$ . In $ϕ$ coordinates, $L (ϕ) = (10 ϕ - 1)^{2} /2$ , so $L^{''} (ϕ) = 100$ . The same predictor has the same performance on any fixed evaluation set. Raw coordinate sharpness alone therefore cannot establish worse generalization. This does not prove every normalized sharpness measure is useless.

Establish the comparison

Record data splits, architecture, augmentations, initialization, batch, precision, schedules, decay and the resource budget. Give both optimizer families a declared tuning opportunity. Select on validation evidence, then compare on independent evaluation with a planned repetition and uncertainty rule.

Claim	Evidence needed	Alternative explanation	Discriminating check
One optimizer generalizes better	Held-out results under the comparison contract	Unequal tuning or augmentation	Match opportunity and repeat
The gap is caused by sharpness	A defined measure with coordinate controls	Reparameterization changes curvature	Test equivalent predictors
AdamW fixes the entire gap	Workload-specific tuned comparisons	Schedule or decay changed too	Separate those interventions

Change the comparison

1. Limit elapsed time

New task: choose a procedure under a one-hour training budget, not equal steps.

Picture and change: compare quality at the deadline and record steps, tokens and overhead. Match tuning opportunity as well as final-run time.

Check: hypothetical step times of $0.25$ and $0.5$ seconds permit $14400$ versus $7200$ steps, ignoring overhead. Quality after $10000$ steps alone cannot decide the one-hour question.

Use when: time or compute limits the real decision.

2. Separate adaptive decay from independent shrinkage

New task: determine whether a comparison changes regularization as well as optimizer family.

Picture and change: a coupled penalty adds $λ θ$ before adaptive scaling. Decoupled decay shrinks parameters independently of that scaling. Even holding a diagonal step scale $D$ fixed, the updates differ:

h e t a_{coupled} = θ - η D (g + λ θ), h e t a_{decoupled} = (1 - η λ) θ - η D g .

Check: with $θ = (1, 1)$ , $g = 0$ , $D = diag (1, 0.1)$ and $η = λ = 0.1$ , results are $(0.99, 0.999)$ versus $(0.99, 0.99)$ . In Adam, the coupled term also enters the accumulated moments.

Use when: isolating decay effects. Retune appropriate strengths before attributing the gap to moment normalization.

Unequal search can create the gap

One family receiving a large rate sweep while another receives one default does not isolate optimizer quality. AdamW changes the update, but does not guarantee better generalization everywhere.

Try interpreting equal training loss

Does equal training loss plus a sharper-looking minimum establish worse generalization?

Check your answer

No. Measure independent predictive performance and define the curvature measure and its coordinate dependence.

Workload and batch regime

Vision, language and large-batch workloads require specific comparison evidence. Neither optimizer family wins by definition, and Adam is not identical to AdamW. Decoupled weight decay changes adaptive regularization; its benchmark results are not universal rankings. LAMB adds layer-wise adaptation for large-batch training under a particular recipe. Its existence does not show that ordinary Adam or SGD always fails above a fixed batch size. Gradient-noise and implicit-bias explanations need defined assumptions and tests, not just a valley illustration.

Common follow-up

For an AdamW schedule, state the intended horizon, warmup, peak, final rate and selection procedure. Warmup-plus-cosine and stable-plus-final-decay are alternatives to test under matched budgets. Fixed warmup fractions and rate multipliers are not universal guarantees.

Why does Adam sometimes generalize worse than SGD?

Practice before you read