Dropout - Neural Networks

Dropout (inverted)

What you'll build

The forward pass of inverted dropout: in training, zero each activation independently with probability 1 - keep_prob and scale the survivors by 1 / keep_prob; at inference, return the input unchanged. Also return the mask, which the backward pass needs.

Background

Dropout (Srivastava et al., 2014) is a regulariser: randomly removing units prevents them from co-adapting, a bit like training an ensemble of thinned networks that share weights. The "inverted" variant scales during training so that the expected value of each activation is unchanged, which means inference needs no rescaling at all. This is what nn.Dropout does.

Algorithm

training:
    mask = (rand(X.shape) < keep_prob) / keep_prob     # entries are 0 or 1/keep_prob
    out  = X * mask
    return out, mask
inference:
    return X, None

E[out] = keep_prob * (X / keep_prob) + (1 - keep_prob) * 0 = X.

Inputs

• X: array of any shape
• keep_prob: probability of keeping a unit, in (0, 1] (note: nn.Dropout(p) takes the drop probability p = 1 - keep_prob)
• training: True for training mode, False for inference

Expected return format

Return the tuple (out, mask):

• training: out has the shape of X and equals X * mask; mask has the shape of X with entries 0 or 1 / keep_prob
• inference: (X, None)

Edge cases and conventions

• Draw the mask with np.random.rand (uniform in [0, 1)) and keep where rand < keep_prob. The tests seed NumPy, check the structure of the mask and statistics over many units, not one specific mask.
• keep_prob = 1.0 keeps everything: out equals X.
• Don't modify X in place.

Constraints

NumPy only, vectorised.

Follow-up

• The backward pass is dX = dout * mask. Why is storing the scaled mask convenient?
• Why is dropout rarely used together with BatchNorm in CNNs, and why do many LLMs train with dropout 0?

Input: dropout_forward(np.array([[1, 2, 3, 4]]), keep_prob=0.5, training=False)
Output: (array([[1, 2, 3, 4]]), None)

Input: dropout_forward(np.array([[1.0, 2.0, 3.0, 4.0]]), keep_prob=0.5, training=True)
Output: e.g. (array([[2., 0., 6., 0.]]), array([[2., 0., 2., 0.]]))

import numpy as np

def dropout_forward(X, keep_prob=0.5, training=True):
    """
    Inverted dropout forward pass.

    Args:
        X: Input array (any shape)
        keep_prob: Probability of keeping each unit, in (0, 1]
        training: True for training mode, False for inference

    Returns:
        out: X * mask in training; X unchanged in inference
        mask: Array like X with entries 0 or 1/keep_prob in training; None in inference
    """
    # Your code here
    pass