Label Smoothing - PyTorch Basics

Label Smoothing

What you'll build: label_smoothing_loss(logits, targets, num_classes, smoothing), cross-entropy against softened targets, averaged over the batch.

Background

With one-hot targets, cross-entropy keeps rewarding the model for pushing the correct logit further above the others, which leads to over-confident, poorly calibrated predictions. Label smoothing (Szegedy et al., 2016) moves a fraction eps of the target mass onto all classes uniformly, so the optimum is a finite logit gap. It is standard in image classification and machine translation, and built into PyTorch as F.cross_entropy(..., label_smoothing=eps).

Formula

For K = num_classes and smoothing eps:

smooth = (1 - eps) * one_hot(target) + eps / K           # shape (batch, K), each row sums to 1
loss_i = -sum_k smooth[i, k] * log_softmax(logits_i)[k]
loss   = mean over the batch of loss_i

The correct class gets 1 - eps + eps / K, every other class eps / K (this matches PyTorch's definition). With eps = 0 it is ordinary cross-entropy.

Inputs

• logits: float array (batch, K) of raw scores.
• targets: integer array (batch,) with values in [0, K).
• num_classes: K.
• smoothing: eps in [0, 1) (default 0.1).

Expected Return Format

A Python float.

Edge Cases and Conventions

• Compute log_softmax stably (subtract the row max) instead of taking log of a softmax.
• Smoothing raises the loss of a confident correct prediction, because the target now also wants some mass on the wrong classes.

Constraints

• Vectorised over the batch.
• PyTorch track: F.log_softmax and F.one_hot are fine; don't use the label_smoothing argument of F.cross_entropy or nn.CrossEntropyLoss.

Follow-up

• Show that the loss can be rewritten as (1 - eps) * CE + eps * mean_k(-log p_k). What is the second term?
• Why can label smoothing hurt knowledge distillation from the smoothed model?

Input: label_smoothing_loss(np.array([[2.0, 1.0, 0.1]]), np.array([0]), 3, 0.1)
Output: 0.5137

Input: label_smoothing_loss(np.array([[2.0, 1.0, 0.1]]), np.array([0]), 3, 0.0)
Output: 0.4170

import numpy as np

def label_smoothing_loss(logits, targets, num_classes, smoothing=0.1):
    """
    Cross-entropy loss with label smoothing, averaged over the batch.

    Args:
        logits: Raw scores (batch, num_classes)
        targets: Integer labels (batch,)
        num_classes: Total number of classes K
        smoothing: Epsilon, the target mass spread uniformly over the K classes

    Returns:
        Scalar loss (float)
    """
    # Your code here
    pass