ROC Curve & AUC (with Ties) - Model Evaluation

ROC Curve & AUC (with Ties)

What you'll build

roc_curve_auc(y_true, scores): the ROC curve at every distinct score threshold and the area under it, with tied scores handled the way scikit-learn handles them.

Background

A classifier that outputs scores becomes a yes/no decision with a threshold: predict positive when score >= t. The ROC curve shows every trade-off between catching positives and raising false alarms, and its area (AUC) summarises ranking quality in one threshold-free number: 1.0 is a perfect ranking, 0.5 is random.

Formula

For a threshold t, with P positives and N negatives:

TPR(t) = #{score >= t and y = 1} / P        (recall)
FPR(t) = #{score >= t and y = 0} / N

Thresholds: +inf first (nothing predicted positive: the point (0, 0)), then every distinct score in decreasing order. The last threshold is the smallest score, where everything is predicted positive: (1, 1).

AUC by the trapezoidal rule over consecutive curve points:

AUC = sum_i (FPR[i+1] - FPR[i]) * (TPR[i+1] + TPR[i]) / 2

With ties handled as below, this equals the normalised Mann-Whitney U statistic, the probability that a random positive outscores a random negative, counting ties as one half:

AUC = (1 / (P * N)) * sum_{i in pos} sum_{j in neg} ( [s_i > s_j] + 0.5 * [s_i == s_j] )

Inputs

• y_true: binary labels (0/1), shape (n,), with both classes present
• scores: real-valued scores, shape (n,); higher means more likely positive

Expected Return Format

Return a dictionary with these keys (T = number of distinct scores):

• 'thresholds': float array of shape (T + 1,): [inf, s_max, ..., s_min]
• 'fpr': float array of shape (T + 1,), from 0.0 to 1.0, non-decreasing
• 'tpr': float array of shape (T + 1,), from 0.0 to 1.0, non-decreasing
• 'auc': Python float

Edge cases and conventions

• Tied scores form a single threshold. All the tied samples cross it together, so the curve takes one diagonal step instead of a staircase. Never break ties by sample order: that would make the AUC depend on how the data happens to be sorted.
• This matches sklearn.metrics.roc_curve(y_true, scores, drop_intermediate=False) (recent versions also start with inf), and the AUC matches sklearn.metrics.roc_auc_score.

Constraints

• NumPy only (no sklearn). Vectorise the counts.

Follow-up

• Compute the AUC in O(n log n) from ranks alone (average ranks for ties).
• Why can ROC-AUC look excellent on a heavily imbalanced problem where precision is poor?

Input: y_true = [0, 0, 1, 1], scores = [0.1, 0.4, 0.35, 0.8]
Output: thresholds [inf, 0.8, 0.4, 0.35, 0.1], fpr [0, 0, 0.5, 0.5, 1], tpr [0, 0.5, 0.5, 1, 1], auc 0.75

Input: y_true = [0, 1], scores = [0.5, 0.5]
Output: fpr [0, 1], tpr [0, 1], auc 0.5

import numpy as np

def roc_curve_auc(y_true, scores):
    """
    ROC curve over all distinct thresholds, and AUC by the trapezoidal rule.

    Args:
        y_true: Binary labels (0/1), shape (n,)
        scores: Real-valued scores, shape (n,); higher means more positive

    Returns:
        dict with 'thresholds', 'fpr', 'tpr' (each shape (T+1,)) and 'auc' (float)
    """
    # Your code here
    pass