Sinusoidal Positional Encoding - Transformer Architecture

Sinusoidal Positional Encoding

What you'll build

The fixed position table from "Attention Is All You Need": one d_model-dimensional vector per position, made of sines and cosines at geometrically spaced frequencies. It is added to the token embeddings.

Background

Attention is permutation-invariant: shuffle the tokens and each output is shuffled the same way. Without position information, "dog bites man" and "man bites dog" look identical. The sinusoidal table gives every position a unique pattern, and because sin/cos at a fixed frequency rotate as the position advances, PE[pos + k] is a linear function of PE[pos] for every offset k. Modern LLMs mostly use RoPE (later in this section) instead, which applies the same frequencies as rotations inside attention.

Formula

For position pos and pair index i = 0, 1, ..., d_model/2 - 1:

omega_i         = 1 / 10000^(2i / d_model)
PE[pos, 2i]     = sin(pos * omega_i)
PE[pos, 2i + 1] = cos(pos * omega_i)

Column pair (2i, 2i+1) shares the frequency omega_i; frequencies go from 1 (fast) down to about 1/10000 (slow).

Inputs

• max_len: int, number of positions (rows), positions are 0 .. max_len - 1
• d_model: even int, embedding width

Expected Return Format

PE: float array (max_len, d_model), values in [-1, 1].

Edge cases and conventions

• Even columns are sines, odd columns are cosines (interleaved, not first half / second half).
• Row 0 is [0, 1, 0, 1, ...] because sin(0) = 0 and cos(0) = 1.
• Computing omega_i as np.exp(-2i * ln(10000) / d_model) is equivalent and avoids huge powers.

Constraints

NumPy only, vectorised (an outer product of positions and frequencies, no Python loops).

Follow-up

• Why does PE[pos + k] = R_k @ PE[pos] for a block-diagonal rotation R_k?
• Learned position embeddings (GPT-2, used in Mini GPT) vs sinusoidal vs RoPE: which ones extrapolate past the training length?

Input: positional_encoding(3, 4)
Output: [[0.0, 1.0, 0.0, 1.0], [0.8415, 0.5403, 0.01, 1.0], [0.9093, -0.4161, 0.02, 0.9998]]

Input: positional_encoding(100, 512).shape
Output: (100, 512)

import numpy as np

def positional_encoding(max_len, d_model):
    """
    Generate sinusoidal positional encodings.

    Args:
        max_len: Number of positions (rows)
        d_model: Model dimension (even)

    Returns:
        PE: (max_len, d_model); PE[pos, 2i] = sin(pos * w_i),
            PE[pos, 2i+1] = cos(pos * w_i), w_i = 1 / 10000^(2i / d_model)
    """
    # Your code here
    pass