Expected Calibration Error - Model Evaluation

Expected Calibration Error

What you'll build

expected_calibration_error(probs, labels, n_bins=10): bin predictions by confidence and measure the average gap between confidence and accuracy, returning the per-bin statistics too (the data behind a reliability diagram).

Background

A classifier is calibrated if, among the predictions it makes with confidence 0.8, about 80% are correct. Calibrated probabilities matter whenever a downstream decision uses the number itself: thresholds, expected-cost calculations, abstaining when unsure. Modern neural networks are often over-confident, and ECE measures by how much.

Formula

For each sample i with predicted class probabilities probs[i] (shape (C,)):

confidence_i = max_c probs[i, c]
prediction_i = argmax_c probs[i, c]          (ties: the lowest class index, as np.argmax)
correct_i    = 1 if prediction_i == labels[i] else 0

Split [0, 1] into M = n_bins equal-width bins with edges np.linspace(0, 1, M + 1). Bin m is the half-open interval (m/M, (m+1)/M] (bin 0 also includes 0.0). Then

count_m = |B_m|
acc_m   = mean(correct_i     for i in B_m)     (0 for an empty bin)
conf_m  = mean(confidence_i  for i in B_m)     (0 for an empty bin)
ECE     = sum_m (count_m / n) * |acc_m - conf_m|

Inputs

• probs: predicted probabilities, shape (n, C), rows sum to 1
• labels: true class indices, shape (n,)
• n_bins: number of equal-width bins M (>= 1)

Expected Return Format

Return a dictionary with these keys:

• 'ece': Python float in [0, 1]
• 'bin_acc': float array of shape (M,): accuracy per bin
• 'bin_conf': float array of shape (M,): mean confidence per bin
• 'bin_count': int array of shape (M,): number of samples per bin

Edge cases and conventions

• Edges go to the bin below: with M = 4, a confidence of exactly 0.5 is in bin 1 = (0.25, 0.5] and 1.0 is in bin 3. In NumPy the bin index is np.searchsorted(edges[1:-1], confidence, side='left'), the number of interior edges strictly below the confidence.
• Empty bins report acc = conf = 0 and contribute nothing to the ECE.
• ECE is 0 for a perfectly calibrated model (on the binned data).

Constraints

• NumPy only, vectorised (np.searchsorted, np.bincount with weights).

Follow-up

• ECE depends on the number of bins. What are the alternatives (adaptive equal-mass bins, the Brier score)?
• How does temperature scaling fix over-confidence without changing the model's accuracy?

Input: expected_calibration_error(np.array([[0.9, 0.1], [0.8, 0.2], [0.4, 0.6], [0.3, 0.7]]), np.array([0, 1, 1, 0]), n_bins=2)
Output: {'ece': 0.25, 'bin_acc': [0.0, 0.5], 'bin_conf': [0.0, 0.75], 'bin_count': [0, 4]}

Input: expected_calibration_error(np.array([[0.75, 0.25]] * 4 + [[0.5, 0.5]] * 2), np.array([0, 0, 0, 1, 0, 1]), n_bins=4)
Output: {'ece': 0.0, 'bin_acc': [0.0, 0.5, 0.75, 0.0], 'bin_conf': [0.0, 0.5, 0.75, 0.0], 'bin_count': [0, 2, 4, 0]}

import numpy as np

def expected_calibration_error(probs, labels, n_bins=10):
    """
    Expected calibration error with equal-width (lo, hi] confidence bins.

    Args:
        probs: Predicted class probabilities, shape (n, C)
        labels: True class indices, shape (n,)
        n_bins: Number of equal-width bins over [0, 1]

    Returns:
        dict with 'ece' (float), 'bin_acc' (M,), 'bin_conf' (M,), 'bin_count' (M,) int
    """
    # Your code here
    pass