Instance Normalization - PyTorch Basics

Instance Normalization

What you'll build: instance_norm(x, gamma, beta, eps) for images (N, C, H, W): normalise every channel of every sample over its spatial positions, then scale and shift per channel.

Background

Instance Normalization (Ulyanov et al., 2016) was introduced for style transfer: the mean and variance of a feature map carry much of an image's "style" (contrast, colour cast), so removing them per image and per channel makes the network's output independent of the input's contrast. It is GroupNorm with one channel per group and is common in GANs and image-to-image models.

Formula

For each sample n and channel c, over the H * W positions:

mean[n, c] = mean over (h, w) of x[n, c, h, w]
var[n, c]  = mean over (h, w) of (x[n, c, h, w] - mean[n, c])²       (population variance)
out[n, c, h, w] = gamma[c] * (x[n, c, h, w] - mean[n, c]) / sqrt(var[n, c] + eps) + beta[c]

Which Axes Does Each Norm Reduce Over?

Norm	Statistics over	One mean per
BatchNorm	(N, H, W)	channel
LayerNorm	(C, H, W)	sample
InstanceNorm	(H, W)	(sample, channel)
GroupNorm	(C/G, H, W)	(sample, group)

Inputs

• x: float array (N, C, H, W).
• gamma, beta: float arrays (C,).
• eps: small positive float (default 1e-5).

Expected Return Format

An array of shape (N, C, H, W). With gamma = 1, beta = 0, every out[n, c] has mean 0 and variance close to 1.

Constraints

• Vectorised; no loops over samples or channels.
• PyTorch track: don't call F.instance_norm or nn.InstanceNorm2d.

Follow-up

• Why does InstanceNorm hurt classification even though it helps style transfer?
• PyTorch's nn.InstanceNorm2d has affine=False by default. What does that change?

Input: instance_norm(np.array([[[[1.0, 2.0], [3.0, 4.0]]]]), np.ones(1), np.zeros(1))
Output: array([[[[-1.3416, -0.4472], [ 0.4472,  1.3416]]]])

Input: instance_norm(np.random.randn(2, 3, 4, 4), np.ones(3), np.array([1.0, 2.0, 3.0])).mean(axis=(2, 3))
Output: array([[1., 2., 3.], [1., 2., 3.]])

import numpy as np

def instance_norm(x, gamma, beta, eps=1e-5):
    """
    Instance normalization for 4D input.

    Args:
        x: Input (N, C, H, W)
        gamma: Scale parameter (C,)
        beta: Shift parameter (C,)
        eps: Numerical stability constant

    Returns:
        Normalized output (N, C, H, W)
    """
    # Your code here
    pass