PPO Clipped Objective - Reinforcement Learning

PPO Clipped Objective

What you'll build

The clipped surrogate objective L^CLIP of Proximal Policy Optimization for a batch of sampled actions.

Background

After collecting a batch with the old policy π_old, PPO takes several gradient steps on it. Without a brake, those steps can push the new policy far from π_old, where the old samples no longer tell you anything. PPO clips the probability ratio so that moving it beyond [1 - ε, 1 + ε] earns no extra objective. It is the default on-policy deep RL algorithm and the optimizer in classic RLHF.

Formula

ratio_t = π_θ(a_t|s_t) / π_old(a_t|s_t)
L^CLIP  = mean_t [ min( ratio_t · A_t,  clip(ratio_t, 1-ε, 1+ε) · A_t ) ]

How the min behaves:

• A > 0 (good action): the objective grows with the ratio only up to 1 + ε.
• A < 0 (bad action): the objective stops improving once the ratio falls below 1 - ε.
• When the change makes things worse (for example ratio > 1 + ε with A < 0), the unclipped term is smaller, so the min keeps it: the objective stays pessimistic.

Inputs

• old_probs, new_probs: (N,) probabilities of the taken actions under π_old and π_θ, in (0, 1]
• advantages: (N,) advantage estimates
• epsilon: clip range, typically 0.2

Expected return format

A Python float: L^CLIP (to maximize; the training loss is its negative), rounded to 2 decimals.

Edge cases and conventions

• Compute the ratio as new_probs / (old_probs + 1e-8).
• Average with the plain mean over the N samples.

Constraints

NumPy only, vectorized.

Follow-up

• Real implementations work with log-probs: ratio = exp(logp_new - logp_old). Why?
• What does "clip fraction" measure, and what is the value-function and entropy part of the full PPO loss?

Input: old_probs = [0.5], new_probs = [0.6], advantages = [1.0], epsilon = 0.2
Output: 1.2

Input: old_probs = [0.5], new_probs = [0.7], advantages = [-1.0], epsilon = 0.2
Output: -1.4

Input: old_probs = [0.5], new_probs = [0.2], advantages = [1.0], epsilon = 0.2
Output: 0.4

import numpy as np

def ppo_clip_objective(old_probs: np.ndarray, new_probs: np.ndarray,
                       advantages: np.ndarray, epsilon: float) -> float:
    """
    Compute the PPO clipped surrogate objective.

    Args:
        old_probs: (N,) pi_old(a|s) of the taken actions
        new_probs: (N,) pi_theta(a|s) of the taken actions
        advantages: (N,) advantage estimates A_t
        epsilon: clipping parameter (e.g. 0.2)

    Returns:
        mean_t min(ratio_t * A_t, clip(ratio_t, 1-eps, 1+eps) * A_t)
        as a float (to maximize), rounded to 2 decimals
    """
    old_probs = np.array(old_probs, dtype=float)
    new_probs = np.array(new_probs, dtype=float)
    advantages = np.array(advantages, dtype=float)

    # Your code here
    pass