Direct Preference Optimization (DPO) - Reinforcement Learning

Direct Preference Optimization (DPO)

What you'll build

The DPO loss: train the policy directly on preference pairs, with no reward model and no RL loop.

Background

RLHF maximizes reward with a KL penalty to a reference model. DPO (Rafailov et al., 2023) shows that the optimum of that problem satisfies r(x, y) = β·log(π_θ(y|x) / π_ref(y|x)) + const. Substituting this implicit reward into the Bradley–Terry loss (see RLHF Reward Model Loss) gives a plain classification loss on log-probabilities. It is simpler and more stable than PPO-RLHF, and widely used for open-weight chat models.

Formula

For each pair, with sequence log-probs (the sum of the response's token log-probs):

Δ_chosen   = log π_θ(y_w|x) - log π_ref(y_w|x)
Δ_rejected = log π_θ(y_l|x) - log π_ref(y_l|x)
L_DPO      = -mean[ log σ( β · (Δ_chosen - Δ_rejected) ) ]

• π_θ: policy being trained; π_ref: frozen reference (usually the SFT model)
• β: strength of the implicit KL constraint (typically 0.1 to 0.5); larger β keeps π_θ closer to π_ref

Inputs

Four (N,) arrays of sequence log-probs (≤ 0): logp_chosen, logp_rejected under π_θ, and ref_logp_chosen, ref_logp_rejected under π_ref; and beta (float > 0).

Expected return format

A Python float: the mean loss, rounded to 3 decimals.

Edge cases and conventions

• At initialization π_θ = π_ref, so every Δ is 0 and the loss is log 2 ≈ 0.693.
• Use a stable log-sigmoid (-np.logaddexp(0, -z)); no eps inside the log.

Constraints

NumPy only.

Follow-up

• DPO can lower the likelihood of both responses as long as the rejected one drops faster. Why can that hurt, and what do variants such as IPO or adding an SFT term do?
• Which quantity would you log as "reward accuracy" during DPO training?

Input: logp_chosen = [-1], logp_rejected = [-2], ref_logp_chosen = [-1.5], ref_logp_rejected = [-1.5], beta = 0.1
Output: 0.644

Input: logp_chosen = [-3, -2], logp_rejected = [-1, -1], ref logps all -2, beta = 0.5
Output: 1.144

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dpo_loss(logp_chosen: np.ndarray, logp_rejected: np.ndarray,
             ref_logp_chosen: np.ndarray, ref_logp_rejected: np.ndarray,
             beta: float) -> float:
    """
    Compute the Direct Preference Optimization (DPO) loss.

    Args:
        logp_chosen: (N,) log pi_theta(y_w|x), sequence log-probs
        logp_rejected: (N,) log pi_theta(y_l|x)
        ref_logp_chosen: (N,) log pi_ref(y_w|x)
        ref_logp_rejected: (N,) log pi_ref(y_l|x)
        beta: KL-strength / temperature parameter

    Returns:
        mean of -log sigmoid(beta * (delta_chosen - delta_rejected)) as a float,
        rounded to 3 decimals
    """
    logp_chosen = np.array(logp_chosen, dtype=float)
    logp_rejected = np.array(logp_rejected, dtype=float)
    ref_logp_chosen = np.array(ref_logp_chosen, dtype=float)
    ref_logp_rejected = np.array(ref_logp_rejected, dtype=float)

    # Your code here
    pass