Precision-Recall Curve and Best-F1 Threshold - Model Evaluation

Precision-Recall Curve and Best-F1 Threshold

What you'll build

pr_curve(y_true, scores): precision, recall and F1 at every distinct score threshold, plus the threshold with the best F1, which is a common way to pick a model's operating point.

Background

When positives are rare (fraud, disease, defects), the precision-recall curve says far more than ROC: it shows how many false alarms you pay for each extra positive found. Teams often pick the deployment threshold by maximising F1 (or another F-beta) on validation data.

Formula

For a threshold t, predict positive when score >= t:

TP(t) = #{score >= t and y = 1}      FP(t) = #{score >= t and y = 0}
P     = #{y = 1}                     FN(t) = P - TP(t)

precision(t) = TP / (TP + FP)
recall(t)    = TP / P
F1(t)        = 2 * TP / (2 * TP + FP + FN)      (equals 2PR / (P + R) when P + R > 0)

Inputs

• y_true: binary labels (0/1), shape (n,)
• scores: classifier scores, shape (n,)

Expected Return Format

Return a dictionary with these keys:

• 'thresholds': list of floats, the distinct scores in descending order
• 'precision', 'recall', 'f1': lists of floats, one per threshold
• 'best_threshold': float, the threshold with the highest F1
• 'best_f1': float, that F1

Edge cases and conventions

• Ties: tied scores enter together: the threshold equal to a tied score counts every sample with that score.
• Every threshold is an observed score, so TP + FP >= 1 and precision is always defined.
• No positive labels (P = 0): recall is set to 1.0 at every threshold, as scikit-learn's precision_recall_curve does, and F1 is 0 (it is 0 whenever TP = 0).
• Best threshold ties: the higher threshold wins (the first one in the descending list).
• Differences from sklearn.metrics.precision_recall_curve: scikit-learn lists thresholds in increasing order and appends a final (precision=1, recall=0) point with no threshold; we use descending order and no extra point.

Constraints

• NumPy only (no sklearn); O(n log n) with a sort and cumulative sums.

Follow-up

• How would you pick a threshold that guarantees precision >= 0.9 with maximum recall?
• Compute average precision (the step-wise area under this curve).

Input: pr_curve(np.array([1, 0, 1, 0, 1, 0, 1, 0]), np.array([0.9, 0.8, 0.7, 0.7, 0.6, 0.4, 0.3, 0.1]))
Output: {'thresholds': [0.9, 0.8, 0.7, 0.6, 0.4, 0.3, 0.1], 'precision': [1.0, 0.5, 0.5, 0.6, 0.5, 0.5714, 0.5], 'recall': [0.25, 0.25, 0.5, 0.75, 0.75, 1.0, 1.0], 'f1': [0.4, 0.3333, 0.5, 0.6667, 0.6, 0.7273, 0.6667], 'best_threshold': 0.3, 'best_f1': 0.7273}

import numpy as np

def pr_curve(y_true, scores):
    """
    Precision, recall and F1 at every distinct score threshold.

    Args:
        y_true: Binary labels (0/1), array (n,)
        scores: Classifier scores, array (n,); predict positive when score >= threshold

    Returns:
        dict with 'thresholds' (descending), 'precision', 'recall', 'f1' (lists of floats),
        'best_threshold' (float) and 'best_f1' (float)
    """
    # Your code here
    pass