Learn · Intermediate

Label smoothing: teaching a classifier not to be certain beyond the evidence

Label smoothing is a regularization method that replaces a perfectly certain class label with a slightly softened target distribution during training. It matters because ordinary cross-entropy training rewards a classifier for pushing the correct answer’s probability toward one and every other answer toward zero, even when the data cannot support that degree of certainty.

Key facts

• With 10 classes and smoothing strength 0.1, a common target gives 0.9 probability to the labeled class and distributes 0.1 across alternatives.
• The model still learns which class is correct; the technique changes how violently it is rewarded for certainty.
• Label smoothing is applied during training, not merely as a display transformation after prediction.
• It is a regularizer, not a guarantee that predicted probabilities are calibrated.

Consider a photo labeled “golden retriever.” A one-hot target says the correct label has probability exactly one and every alternative—labrador, collie, mixed breed—has probability zero. But real labels are noisy, images can be ambiguous, and a dataset’s taxonomy is not the world. A model trained to make every labeled item a courtroom certainty may learn brittle, extremely peaked scores. Label smoothing instead tells it: this example strongly supports golden retriever, but do not act as if the other nine classes are metaphysically impossible.

Formally, ordinary classification uses a target vector with a one at the true class and zeros elsewhere. Label smoothing mixes that vector with a simple prior distribution, usually uniform. If the smoothing coefficient is epsilon, the true class receives 1 - epsilon plus its share of the prior, depending on the convention; other classes receive small positive mass. Libraries differ slightly in their exact formula, so it is worth reading the implementation rather than assuming that “0.1” has identical semantics everywhere. The cross-entropy loss then compares the model’s softmax distribution with this softened target.

Christian Szegedy and colleagues introduced the widely used version in Rethinking the Inception Architecture for Computer Vision. The intuition is regularization: the model is discouraged from making one logit infinitely larger than the rest. That can reduce overfitting, improve top-line accuracy in some classification settings and make representations less tied to a single hard label. It resembles a teacher saying, “This answer is clearly best, but do not learn that every other answer is absurd.”

The technique is especially useful when labels are imperfect, classes are fine-grained, or a downstream system will compare scores across examples. A vision classifier may be uncertain between two dog breeds; a language classifier may face overlapping intent labels. Softened targets can make optimization less eager to memorize arbitrary annotation boundaries. It is also cheap: no extra model, data pass or inference latency is required.

But label smoothing has sharp limits. It assumes the alternatives should receive roughly similar residual probability. That is often wrong. In a medical classifier, confusing two nearby conditions is not like confusing either with a completely unrelated class; in a hierarchical taxonomy, the prior should not be uniform. Smoothing can also harm tasks that need very sharp confidence or knowledge distillation setups where the output distribution carries useful fine-grained information. The analysis in When Does Label Smoothing Help? shows why its benefits are not a universal law.

It is crucial not to confuse smoothing with calibration. Calibration asks whether predictions assigned 80 percent confidence are correct about 80 percent of the time on matching held-out cases. Label smoothing changes training incentives and can improve or worsen that relationship depending on data, architecture and shift. A model can emit less extreme probabilities yet still be systematically overconfident, underconfident or wrong on unfamiliar inputs. Measure reliability curves, expected calibration error and decision cost after training; if needed, apply a post-hoc technique such as temperature scaling on a validation set.

Label smoothing also differs from simply adding noisy labels. The true label remains the dominant target; the small alternative mass is controlled and intentional. It differs from data augmentation, which creates varied inputs, and from dropout or weight decay, which constrain a model through internal computation or parameters. All are regularizers, but they intervene at different places in learning.

For practice, begin modestly—often around 0.05 to 0.1—then test accuracy, calibration and the actual decision metric on untouched validation data. Keep the raw labels for evaluation, since reporting performance against smoothed labels obscures the question users care about. If classes have known relationships, consider a non-uniform prior rather than blindly spreading mass equally. The lesson is not that confidence is bad. It is that a training label is evidence, and good systems should not be paid to pretend the evidence is stronger than it is.

Ground Truth. (2026, September 16). Label smoothing: teaching a classifier not to be certain beyond the evidence. Ground Truth. https://groundtruth.day/learn/label-smoothing.html

@misc{groundtruth:label-smoothing,
  title  = {Label smoothing: teaching a classifier not to be certain beyond the evidence},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/label-smoothing.html}
}

Topics: training · classification · regularization · calibration