Cross-Entropy Loss - Neural Networks

Cross-Entropy Loss

What you'll build

The mean cross-entropy loss between predicted class probabilities and one-hot labels, rounded to 4 decimals.

Background

Cross-entropy is the standard classification loss. Minimising it is maximum likelihood: it is the average negative log-probability the model gave to the correct class. A confident correct prediction costs almost nothing; a confident wrong one costs a lot (-log(0.01) ≈ 4.6).

Formula

L = -(1/m) * sum_i sum_c  Y[i, c] * log(Y_pred[i, c] + eps)        eps = 1e-15

• m: batch size (number of rows)
• Y[i, c]: 1 if example i belongs to class c, else 0
• Y_pred[i, c]: predicted probability of class c for example i
• With one-hot labels, only the true class contributes: L = mean_i(-log(p_i,true)).

Inputs

• Y_pred: probabilities, shape (m, n_classes), each row sums to 1, values in [0, 1]
• Y_true: one-hot labels, shape (m, n_classes)

Expected return format

A Python float: the mean loss over the batch, rounded to 4 decimals with round(loss, 4).

Edge cases and conventions

• Add eps = 1e-15 inside the log so a predicted probability of exactly 0 gives a large finite loss (-log(1e-15) = 34.5388) instead of inf.
• A perfect prediction gives 0.0 (up to the sign of a tiny rounding error).

Constraints

NumPy only, vectorised. Use the probabilities as given (this function does not apply softmax).

Follow-up

• In practice frameworks take logits, not probabilities, and compute logsumexp(z) - z_y, which is stable without an eps. Why is that better than log(softmax(z))?
• With class indices instead of one-hot labels, use -log(p[np.arange(m), y]).

Input: cross_entropy_loss(np.array([[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]]), np.array([[1, 0, 0], [0, 1, 0]]))
Output: 0.2899

Input: cross_entropy_loss(np.array([[0.0, 1.0]]), np.array([[1, 0]]))
Output: 34.5388

import numpy as np

def cross_entropy_loss(Y_pred, Y_true):
    """
    Mean cross-entropy loss for one-hot labels.

    Args:
        Y_pred: Predicted probabilities (m, n_classes)
        Y_true: One-hot true labels (m, n_classes)

    Returns:
        loss: Mean cross-entropy as a float, rounded to 4 decimals
              (use eps = 1e-15 inside the log)
    """
    # Your code here
    pass