Dense Layer Forward Pass (ReLU) - Deep Learning Basics

Dense Layer Forward Pass (ReLU)

What you'll build

The forward pass of one fully-connected (dense) layer with a ReLU activation, for a whole batch at once: dense_forward(X, W, b) = relu(X @ W + b).

Background

A dense layer is the basic unit of an MLP and of the feed-forward block inside every transformer. Each output unit computes a weighted sum of all inputs plus a bias, then a non-linearity. Processing a batch as one matrix product, instead of looping over examples, is what makes it fast on modern hardware.

Formula

Z   = X @ W + b        # (batch, in_dim) @ (in_dim, out_dim) -> (batch, out_dim); b broadcasts over rows
out = max(0, Z)        # ReLU, element-wise

• X: inputs, one example per row
• W: weights; column j holds the weights of output unit j
• b: one bias per output unit, added to every row

Inputs

• X: array of shape (batch, in_dim)
• W: array of shape (in_dim, out_dim)
• b: array of shape (out_dim,)

Expected return format

An array of shape (batch, out_dim) holding relu(X @ W + b).

Edge cases and conventions

• This uses the X @ W layout (weights stored as (in_dim, out_dim)). PyTorch's nn.Linear stores weight as (out_dim, in_dim) and computes X @ weight.T + bias.
• Pre-activations that are exactly 0 or negative give 0.

Constraints

Use one matrix product for the whole batch (no loops over rows); NumPy only.

Follow-up

• How many parameters does this layer have (in_dim * out_dim + out_dim), and what does the forward pass cost (batch * in_dim * out_dim multiply-adds)?
• Write the backward pass: given dOut, what are dW, db and dX? (Done in MLP Forward & Backward Pass.)

Input: dense_forward(np.array([[1, 2]]), np.array([[1, 0], [0, 1]]), np.array([1, 1]))
Output: [[2, 3]]

Input: dense_forward(np.array([[1.0, -1.0], [0.5, 2.0]]), np.array([[1.0, -2.0, 0.5], [0.5, 1.0, -1.0]]), np.array([0.0, 0.5, -0.5]))
Output: [[0.5, 0.0, 1.0], [1.5, 1.5, 0.0]]

import numpy as np

def dense_forward(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    """
    Forward pass of a dense layer with ReLU activation: relu(X @ W + b).

    Args:
        X: Inputs, shape (batch, in_dim)
        W: Weights, shape (in_dim, out_dim)
        b: Bias, shape (out_dim,)

    Returns:
        Activations, shape (batch, out_dim)
    """
    # Your code here
    pass