TD(0) Value Prediction - Reinforcement Learning

TD(0) Value Prediction

What you'll build

The TD(0) update for state values: after one transition (s, r, s'), move V(s) toward the bootstrapped target r + γ·V(s').

Background

Temporal-difference learning updates estimates from other estimates (bootstrapping), so it learns online after every step instead of waiting for the episode to end like Monte Carlo. The TD error δ is the learning signal in actor-critic methods and the building block of GAE.

Formula

target = r + γ · V(s')       if s' is not terminal
target = r                   if done (a terminal state has value 0)
δ      = target - V(s)
V(s)  ← V(s) + α · δ

	TD(0)	Monte Carlo
Target	r + γ·V(s')	full return G_t
Update	every step	end of episode
Bias	biased (uses an estimate)	unbiased
Variance	low	high

Inputs

• V: 1-D array of state values, length n_states
• state, next_state: int indices
• reward, alpha, gamma: floats
• done: bool, True if next_state is terminal

Expected return format

The updated value array (a list also passes), rounded to 2 decimals. Work on a copy; only V[state] changes.

Edge cases and conventions

When done=True, ignore V[next_state] entirely, whatever value is stored there.

Constraints

NumPy only.

Follow-up

• On a fixed batch of data, TD(0) and Monte Carlo converge to different answers. Which one fits the maximum-likelihood Markov model?
• How does TD(λ) interpolate between TD(0) and Monte Carlo?

Input: V = [0, 0, 0], s = 0, r = 1, s' = 1, α = 0.1, γ = 0.9, done = False
Output: [0.1, 0.0, 0.0]

Input: V = [5, 10, 15], s = 1, r = 2, s' = 2, α = 0.1, γ = 0.9, done = False
Output: [5.0, 10.55, 15.0]

import numpy as np

def td0_update(V: np.ndarray, state: int, reward: float,
               next_state: int, alpha: float, gamma: float,
               done: bool = False) -> np.ndarray:
    """
    Perform one TD(0) value update.

    Args:
        V: (n_states,) value estimates
        state: current state s
        reward: reward received r
        next_state: next state s'
        alpha: learning rate
        gamma: discount factor
        done: True if next_state is terminal (its value counts as 0)

    Returns:
        Updated (n_states,) values (a copy), rounded to 2 decimals
    """
    V = np.array(V, dtype=float)
    # Your code here
    pass