Gradient clipping

Summary

Global-norm clipping shortens a finite gradient by one common factor, preserving its direction when the factor is positive. For threshold $c\ge 0$, use $g^{\prime }=g\min (1,c/\Vert g{\Vert }_2)$, leaving a zero vector unchanged. This caps the gradient norm, not every optimizer’s eventual update.

The fixture has $g=(6,8)$, norm ten, and threshold five. Scaling by one-half produces $(3,4)$ with norm five. Ordinary SGD with learning rate $0.1$ moves by $(-0.3,-0.4)$, norm $0.5$.

Among vectors inside the radius-$c$ Euclidean ball, radial projection minimizes distance to $g$. Cauchy-Schwarz makes the nearest nonzero candidate parallel to $g$; its permitted length is $\min (\Vert g\Vert ,c)$. This geometric guarantee neither diagnoses a loss spike nor makes a nonfinite gradient usable. Check finiteness before stepping; infinity multiplied by zero does not produce a repaired direction.

Two flavors

Global-norm clipping (the standard)

Compute the norm over the declared optimizer parameter set after its intended accumulation and synchronization:

Use that same factor for every included tensor. For sharded gradients, aggregate squared norms over unique shards; replicated data-parallel copies are not additional parameter coordinates.

Per-parameter clipping

Separate tensor caps change relative scaling and generally rotate the concatenated gradient.

Value clipping

Coordinate clipping to $[-5,5]$ turns $(6,8)$ into $(5,5)$, with norm $\sqrt{50}$ and a different direction. That is a coordinate bound, not the radius-five constraint.

How to pick the threshold

Log the preclip norm, nonfinite frequency, clipping frequency, and actual optimizer update norm. Threshold units depend on loss reduction, parameterization, and batch accumulation. Rare clipping may be an intentional safety limit; frequent clipping warrants checking the intended learning-rate and loss-scale interaction. Validate a threshold against the training objective rather than treating an architecture-wide number as universal.

Combined with mixed precision

If loss scaling is used, finish accumulation and unscale once before clipping. Otherwise the threshold applies to scaled units; dividing afterward changes the effective cap. BF16 frequently runs without a loss scaler; format choice alone does not imply one.

Change the Requirement

Bound Parameter Movement

New task: Cap the final optimizer displacement rather than its input gradient.

Picture and change: Inspect momentum, preconditioning, learning rate, and decay before choosing an update-level constraint.

Check: With velocity $v=(100,0)$ and rule $v^{\prime }=0.9v+g^{\prime }$, the fixture gives $(93,4)$, not a norm-five update.

Use when: The requirement concerns actual parameter movement. The bound $\Vert \Delta \theta \Vert \le \eta c$ applies to ordinary SGD without those additional transformations.

Accumulate Before Clipping

New task: Combine several microbatches into one optimizer step under a declared summed-loss convention.

Picture and change: Sum contributions first, then clip the combined gradient.

Check: Scalar contributions $10$ and $-9$ sum to $1$; clipping each to five before adding instead gives zero.

Use when: Multiple backward calls represent one update. For mean loss, apply the correct sample-count normalization; averaging unequal batch means defines a different gradient.