Label smoothing

Summary

Label smoothing (Szegedy et al., 2016) replaces the hard one-hot target $y$ with $\tilde{y} = (1 - ε) \cdot y + ε / K$ , where $K$ is the number of classes and $ε$ is a small smoothing constant (typically 0.1). The cross-entropy loss is computed against $\tilde{y}$ .

With one-hot targets, cross-entropy approaches its lower bound of zero as the model becomes confident. It can always reduce loss further by making the correct logit larger; reaching the boundary $p_{c} = 1$ would require an infinite gap between the correct and incorrect logits. This pushes the model toward arbitrarily large logit gaps and overconfident predictions, which are poorly calibrated.

Label smoothing instead creates a finite optimum and forces the model to maintain non-zero probability on incorrect classes. The effects:

Better calibration: predicted probabilities track empirical accuracy more closely.
Slightly better generalization on most classification benchmarks.
Standard in transformer training: original “Attention Is All You Need” used $ε = 0.1$ ; LLM pretraining occasionally uses it.

The mechanism

For a classification problem with $K$ classes and true class $c$ :

Hard target: $y_{i} = 1$ if $i = c$ , else 0.
Smoothed target: $\tilde{y}_{i} = 1 - ε + ε / K$ if $i = c$ , else $ε / K$ .

Cross-entropy with smoothed targets:

L = - i \sum \tilde{y}_{i} lo g p_{i} = - (1 - ε) lo g p_{c} - \frac{ε}{K} i \sum lo g p_{i} .

The first term is the standard cross-entropy; the second is an entropy-like penalty that pulls the predicted distribution toward uniform.

Equivalent view: the optimal $p_{c}$ for label smoothing is $1 - ε + ε / K$ , not 1. The model has no incentive to push the correct logit beyond what produces this target probability.

Learning objective

Uniform mixing leaves target mass 0.92, not 0.90

Derivation

Read it this way: follow the solid curve right: a one-hot target keeps rewarding higher confidence, and its zero-loss boundary requires an unbounded correct-versus-incorrect logit gap. The dashed curve stops at p_c = 0.92, where the prediction matches the smoothed target [0.92, 0.02, 0.02, 0.02, 0.02]. Push farther right and the loss rises because the wrong classes fall below their non-zero targets. This is an original calculation from the formulation in Szegedy et al. (2016).

Check: Why is the optimum 0.92, not 0.90, for K=5 and epsilon=0.1 in this convention?

Read it this way: For five classes and smoothing 0.1, the true target is 0.9+0.1/5=0.92. The other four classes receive 0.02 each.
Does the true class also receive its share of the uniform mass?

When to use

Language modeling or image classification: compare the declared smoothing convention and strength with unsmoothed training under matched validation.
Distillation: teacher probabilities already supply soft targets; extra smoothing is a separate intervention, not automatically necessary or redundant.
Retrieval / contrastive learning: assess ranking and representation quality under the actual candidate/label semantics instead of transferring a classification default.

Side effects

Calibration and accuracy can change in either direction. Smoothing changes the training target; it does not prove probability-frequency agreement on deployment data.
Representation and distillation effects depend on the task. Studies such as Müller et al. (2019) motivate checking teacher information and embedding geometry, not declaring one outcome universal.
Keep the evaluation target explicit. Report ordinary held-out likelihood and task metrics separately from the smoothed training objective; see the cross-entropy mechanism.

Common pitfalls

Stacking target changes without testing them. Mixup/CutMix and smoothing have different constructions; their combination needs an ablation rather than automatic approval or rejection.
Using on a regression problem. Label smoothing is for categorical cross-entropy; it has no meaning for MSE.
Treating a coefficient as universal. Include zero and task-appropriate candidates in validation; the class count and smoothing convention affect what any chosen value means.
Forgetting to disable for eval-only metrics. Loss numbers with label smoothing are not directly comparable to one-hot loss numbers.