Q-Learning Update - Reinforcement Learning

Q-Learning Update

What you'll build

One step of tabular Q-learning: move Q(s, a) toward the Bellman optimality target after observing a transition (s, a, r, s').

Background

Q-learning (Watkins, 1989) learns the optimal action values Q* without a model of the environment. It is off-policy: the target uses the best next action, max_a' Q(s', a'), whatever the behaviour policy (e.g. ε-greedy) will actually do next. DQN is this same update with a neural network in place of the table.

Formula

target = r + γ · max_a' Q(s', a')
δ      = target - Q(s, a)              (TD error)
Q(s, a) ← Q(s, a) + α · δ

• α: learning rate in (0, 1]
• γ: discount factor in [0, 1] Only the entry Q[s, a] changes.

Inputs

• Q: (n_states, n_actions) Q-table
• state, action, next_state: int indices
• reward: float
• alpha, gamma: floats

Expected return format

The updated (n_states, n_actions) Q-table as a float array (a nested list also passes), rounded to 1 decimal. Don't modify the input in place; work on a copy.

Edge cases and conventions

• This problem has no done flag. To model a terminal transition, pass gamma = 0 (the target is then just r).
• The max is over the next state's row, including negative values: if every Q(s', ·) is negative, the max is negative.

Constraints

NumPy only; no loops needed.

Follow-up

• Why does the max make Q-learning overestimate values, and how does Double Q-learning fix it?
• What conditions on α and exploration guarantee convergence to Q*?

Input: Q = [[0, 0], [0, 0]], s = 0, a = 1, r = 1, s' = 1, α = 0.1, γ = 0.9
Output: [[0.0, 0.1], [0.0, 0.0]]

Input: Q = [[1, 2], [3, 4]], s = 0, a = 0, r = 5, s' = 1, α = 0.5, γ = 0.9
Output: [[4.8, 2.0], [3.0, 4.0]]

import numpy as np

def q_learning_update(Q: np.ndarray, state: int, action: int,
                      reward: float, next_state: int,
                      alpha: float, gamma: float) -> np.ndarray:
    """
    Perform one Q-learning update.

    Args:
        Q: (n_states, n_actions) Q-table
        state: current state index s
        action: action taken a
        reward: reward received r
        next_state: next state index s'
        alpha: learning rate
        gamma: discount factor

    Returns:
        Updated (n_states, n_actions) Q-table (a copy), rounded to 1 decimal
    """
    Q = np.array(Q, dtype=float)
    # Your code here
    pass