ReLU Activation - Deep Learning Basics

ReLU Activation

What you'll build

The Rectified Linear Unit, relu(x), applied element-wise to an array of any shape. It is the default non-linearity in the hidden layers of MLPs, CNNs and transformer feed-forward blocks.

Background

Without a non-linearity, a stack of linear layers collapses into a single linear map, so depth buys nothing. ReLU is the simplest non-linearity that works well: it is cheap, it does not saturate for positive inputs (its gradient there is exactly 1, which helps against vanishing gradients), and it makes activations sparse.

Formula

relu(x)  = max(0, x)           element-wise
relu'(x) = 1 if x > 0 else 0   (the value at x = 0 is a convention; frameworks use 0)

Inputs

• x: NumPy array of any shape (ints or floats).

Expected return format

An array with the same shape as x: negative entries become 0, the rest are unchanged. Keep the input's dtype (an int array stays int).

Edge cases and conventions

• relu(0) = 0.
• Tiny negatives such as -1e-10 become exactly 0.

Constraints

Vectorise: no Python loops over elements.

Follow-up

• What is the "dying ReLU" problem, and how do Leaky ReLU and GELU address it? (You implement both in PyTorch Basics.)
• Why does ReLU pair with He initialization rather than Xavier? (See Xavier/He Weight Initialization in Neural Networks.)

Input: relu(np.array([-2, -1, 0, 1, 2]))
Output: [0, 0, 0, 1, 2]

Input: relu(np.array([[-1.5, 2.0], [3.0, -4.0]]))
Output: [[0.0, 2.0], [3.0, 0.0]]

import numpy as np

def relu(x: np.ndarray) -> np.ndarray:
    """
    Apply the ReLU activation element-wise.

    Args:
        x: Input array of any shape

    Returns:
        Array of the same shape and dtype with max(0, x) applied element-wise
    """
    # Your code here
    pass