How do you choose a learning rate?

Choose a learning-rate policy by comparing update behavior and validating a complete training schedule rather than a single sweep result.

Problem

Context. The exact one-dimensional SGD fixture below isolates learning-rate behavior before considering noisy, scheduled training.

Task. Compare the three constant rates, explain convergence or divergence, and propose a controlled learning-rate selection procedure. Describe how warmup, decay, optimizer choice, and intended run length affect the final comparison.

Constraints. Use the stated loss, gradient, and starting point for the fixture; do not add momentum or stochastic noise. A short range test proposes candidates rather than proving the best full-run schedule.

Use plain SGD on $L (w) = 5 w^{2}$ , starting at $w_{0} = 1$ with exact gradient $10 w$ . Compare constant rates $0.1$ , $0.2$ and $0.25$ . Which rates reduce the loss, and what changes when the intended run uses a schedule?

Candidate first, controlled run second

Choose the useful descent region, not the unstable end of the sweep

Derivation

the first plot proposes a rate before instability. The second assigns that candidate to a schedule. These qualitative curves are not measured results or proof of an optimal peak.

Check: Which LR from the sweep becomes the schedule peak? Why is the unstable endpoint not the choice?

Read it this way: For loss x squared, the update multiplies x by one minus twice the learning rate. These three-step fixtures contract, oscillate, or grow. A range test still requires observed task evidence.
Does crossing the minimum necessarily mean the iterations diverge?

Range-test references: Smith (2017) and the fastai learning-rate finder documentation.

Calculate the updates

For plain SGD, $w_{t + 1} = w_{t} - η \nabla L (w_{t}) = (1 - 10 η) w_{t}$ . Contraction requires $∣1 - 10 η ∣ < 1$ , so $0 < η < 0.2$ .

Rate	$w_{0}, w_{1}, w_{2}$	$L (w_{0}), L (w_{1}), L (w_{2})$	Decision
$0.1$	$1, 0, 0$	$5, 0, 0$	Reaches the minimum
$0.2$	$1, - 1, 1$	$5, 5, 5$	Oscillates without progress
$0.25$	$1, - 1.5, 2.25$	$5, 11.25, 25.3125$	Diverges

Finite values alone do not establish progress. The boundary rate $0.2$ never improves this loss. This calculation is not an Adam benchmark or a universal neural-network bound.

Test a candidate

Record the optimizer, loss reduction, batch, initialization, precision and intended training length. Use a disposable run to increase rates while recording loss and update behavior. For $N \geq 2$ sweep steps, a geometric sweep can use

η_{j} = η_{low} (\frac{η _{high}}{η _{low}})^{j / (N - 1)}, j = 0, \dots, N - 1.

Choose a candidate in sustained descent, with margin before instability. Restore the starting model and optimizer state, then test the intended schedule on controlled runs. Compare representative validation results and stability at the same budget. A sweep changes weights along its path; its points are not independent rate trials.

Change the run

1. Double the curvature

New task: replace the loss by $10 w^{2}$ .

Picture and change: the gradient becomes $20 w$ . Recompute contraction from $∣1 - 20 η ∣ < 1$ .

Check: the interval becomes $0 < η < 0.1$ . The old rate $0.1$ now oscillates without progress.

Use when: loss scaling, parameterization or optimizer changes the effective update.

2. Specify a shorter training horizon

New task: use $S = 1000$ steps, $W = 100$ warmup steps, peak $0.001$ and final rate $0.0001$ .

Picture and change: choose the schedule separately from the peak. At schedule position $t \in [0, S]$ , use

η (t) = {0.001 t / W, 0.0001 + \frac{0.0009}{2} [1 + cos (π \frac{t - W}{S - W})], 0 \leq t \leq W, W < t \leq S .

Check: positions $50$ , $100$ , $550$ and $1000$ give rates $0.0005$ , $0.001$ , $0.00055$ and $0.0001$ .

Use when: the step budget changes. These fractions describe this fixture, not a universal schedule.

Do not continue from a failed sweep

Diverged weights and accumulated Adam moments are not a clean starting state. Reload model, optimizer and schedule state; control data order and randomness for the comparison. Otherwise a failed sweep can invalidate the candidate test.

Try changing the batch

Doubling batch size also changes loss reduction from mean to sum. What must be checked first?

Check your answer

Restore the intended reduction or account for its doubled gradient scale. Only then test a batch-scaling heuristic under controlled conditions.

Optimizer and training regime

Adam uses accumulated, coordinate-dependent moments, so its effective update is not simply plain SGD’s gradient times a rate. Batch-scaling rules require assumptions. Neither linear scaling for SGD nor square-root scaling for Adam is universal. The large-minibatch SGD recipe combines scaling with specific training conditions and warmup.

Fine-tuning must balance adaptation against preserving useful behavior. Compare layer updates and held-out behavior instead of assuming pretrained weights are well-conditioned. Fixed fine-tuning, SFT or preference-training rate ranges do not transfer automatically between models and objectives.

Schedules and diagnostics

Warmup limits early updates; its usefulness and mechanism depend on optimizer, parameterization and model. The Transformer schedule, cosine decay, step decay and one-cycle use different time profiles. Compare them at a stated training budget rather than declaring one always superior.

If every reasonable rate fails, inspect data, initialization, gradient scale, loss implementation and numerical precision. Reducing the rate does not repair incorrect labels or a broken backward pass.

How do you choose a learning rate?

Practice before you read