GELU Activation - PyTorch Basics

GELU Activation

What you'll build: gelu(x), the Gaussian Error Linear Unit with the tanh approximation, applied element-wise.

Background

GELU is the activation in the feed-forward blocks of BERT, GPT-2 and most transformers since. Where ReLU gates its input with a hard step (x * [x > 0]), GELU gates it softly with the probability that a standard normal variable is below x: GELU(x) = x * Φ(x). Small negative inputs leak through slightly, and the function is smooth everywhere, which tends to help optimisation in deep networks.

Formula

The exact form uses the normal CDF Φ. GPT-2 and the original BERT code use this tanh approximation, which is what you implement:

GELU(x) ≈ 0.5 * x * (1 + tanh( sqrt(2/π) * (x + 0.044715 * x³) ))

with sqrt(2/π) ≈ 0.79788.

Key Properties

Property	ReLU	GELU
Value at 0	0	0
Negative inputs	exactly 0	small negative outputs (minimum ≈ -0.17 near x = -0.75)
Large positive x	x	approaches x
Large negative x	0	approaches 0 from below
Smooth at 0	no	yes

Inputs

• x: float array of any shape.

Expected Return Format

An array of the same shape as x, with GELU applied element-wise.

Constraints

• Use the tanh approximation above (the tests expect its values to 4 decimals).
• PyTorch track: build it from tensor ops (torch.tanh); don't call F.gelu or nn.GELU.

Follow-up

• What is F.gelu(x) without approximate='tanh', and how far apart are the two versions?
• How does SiLU / Swish (x * sigmoid(x), next problem) compare?

Input: gelu(np.array([-1.0, 0.0, 1.0]))
Output: array([-0.1588,  0.    ,  0.8412])

Input: gelu(np.array([-3.0, 3.0]))
Output: array([-0.0036,  2.9964])

import numpy as np

def gelu(x: np.ndarray) -> np.ndarray:
    """
    Compute the GELU activation function (tanh approximation).

    Args:
        x: Input array of any shape

    Returns:
        GELU applied element-wise, same shape as input
    """
    # Your code here
    pass