LeakyReLU & Swish - PyTorch Basics

LeakyReLU & Swish

What you'll build: two activations that fix ReLU's main weakness: leaky_relu(x, alpha) and swish(x, beta), both element-wise.

Background

A ReLU unit whose input is negative for every example outputs 0 and gets a gradient of exactly 0, so it can never recover: the dying ReLU problem. LeakyReLU keeps a small slope for negative inputs; Swish (called SiLU in PyTorch when beta = 1) replaces the hard gate with a smooth sigmoid gate. SiLU is the activation inside the SwiGLU feed-forward blocks of LLaMA-style models.

Formulas

LeakyReLU(x) = x           if x > 0
               alpha * x   otherwise          (alpha = 0.01 by default)

Swish(x)     = x * sigmoid(beta * x) = x / (1 + exp(-beta * x))      (beta = 1 by default)

Comparison

Property	ReLU	LeakyReLU	Swish
Gradient for x < 0	0 (can die)	alpha	non-zero
Smooth	no	no	yes
Negative outputs	never	yes, alpha * x	yes, small (minimum ≈ -0.278 near x = -1.28 for beta = 1)
Monotonic	yes	yes	no

Inputs

• x: float array of any shape.
• alpha: slope for non-positive inputs (float, default 0.01).
• beta: sharpness of the Swish gate (float, default 1.0). As beta grows, Swish approaches ReLU.

Expected Return Format

Each function returns an array of the same shape as x.

Constraints

• Vectorised element-wise ops; no Python loops.
• PyTorch track: use torch.where and torch.sigmoid; don't call F.leaky_relu, F.silu, nn.LeakyReLU or nn.SiLU.

Follow-up

• Why is x * sigmoid(x) non-monotonic, and does that matter?
• PReLU learns alpha per channel. How would you add it as an nn.Parameter?

Input: leaky_relu(np.array([-2.0, -1.0, 0.0, 1.0, 2.0]))
Output: array([-0.02, -0.01,  0.  ,  1.  ,  2.  ])

Input: swish(np.array([-1.0, 0.0, 1.0, 2.0]))
Output: array([-0.2689,  0.    ,  0.7311,  1.7616])

import numpy as np

def leaky_relu(x: np.ndarray, alpha: float = 0.01) -> np.ndarray:
    """
    Compute LeakyReLU activation.

    Args:
        x: Input array of any shape
        alpha: Slope for non-positive values (default 0.01)

    Returns:
        LeakyReLU applied element-wise, same shape as x
    """
    # Your code here
    pass


def swish(x: np.ndarray, beta: float = 1.0) -> np.ndarray:
    """
    Compute Swish activation: x * sigmoid(beta * x).

    Args:
        x: Input array of any shape
        beta: Scaling parameter (default 1.0)

    Returns:
        Swish applied element-wise, same shape as x
    """
    # Your code here
    pass