Epsilon-Greedy Policy - Reinforcement Learning

Epsilon-Greedy Policy

What you'll build

The ε-greedy action selector: usually take the action with the highest Q-value, but with probability ε take an exploratory action.

Background

A purely greedy agent can lock onto the first action that looked good and never discover a better one. ε-greedy is the simplest fix for this exploration vs exploitation trade-off and is the default behaviour policy of tabular Q-learning and DQN. In practice ε is annealed, e.g. from 1.0 down to 0.05.

Algorithm

The random draw u ~ Uniform[0, 1) is passed in as random_value so tests are deterministic.

1. If random_value < epsilon: explore and return action floor(random_value · n_actions).
2. Otherwise: exploit and return argmax_a Q(a).

Inputs

• q_values: 1-D array of length n_actions
• epsilon: float in [0, 1]
• random_value: float in [0, 1)

Expected return format

A Python int: the selected action index.

Edge cases and conventions

• The comparison is strict: random_value == epsilon exploits. epsilon = 0 always exploits; epsilon = 1 always explores.
• Ties in the greedy step go to the lowest index (what np.argmax does).
• The explore rule floor(random_value · n_actions) is this problem's deterministic stand-in for "pick a uniformly random action".

Constraints

NumPy only.

Follow-up

• Real code draws a second random number for the exploratory action. Why does reusing random_value bias exploration toward low indices when ε is small?
• Compare with softmax (Boltzmann) exploration and UCB.

Input: q_values = [1.0, 3.0, 2.0], epsilon = 0.1, random_value = 0.5
Output: 1

Input: q_values = [1.0, 2.0, 3.0, 4.0], epsilon = 0.5, random_value = 0.2
Output: 0

import numpy as np

def epsilon_greedy(q_values: np.ndarray, epsilon: float,
                   random_value: float) -> int:
    """
    Select an action with the epsilon-greedy policy.

    Args:
        q_values: (n_actions,) Q-values [Q(a_0), Q(a_1), ...]
        epsilon: exploration probability in [0, 1]
        random_value: pre-drawn uniform number in [0, 1), for deterministic tests

    Returns:
        Selected action index (int)
    """
    q_values = np.array(q_values)
    n_actions = len(q_values)

    # Your code here
    pass