ResNet Basic Block - CNNs & Computer Vision

ResNet Basic Block

What you'll build

residual_block(x, params, stride=1): the "basic block" of ResNet-18/34, with an identity or projection shortcut, in NCHW layout.

Background

Very deep plain CNNs train worse than shallower ones: optimisation, not overfitting, is the problem. ResNet (He et al., 2015) adds a skip connection: each block learns a residual F(x) and outputs ReLU(F(x) + x). If the identity is the best a block can do, it only has to push F to zero, and gradients flow straight through the additions. This made 100+ layer networks trainable, and the same residual idea is in every transformer.

x ──► conv3x3(stride) ─► BN ─► ReLU ─► conv3x3 ─► BN ──► (+) ─► ReLU ─► out
│                                                         ▲
└──────────────── shortcut: identity or 1x1 conv + BN ────┘

Algorithm

1. Main path: conv1 3x3, stride stride, padding 1, C_in -> C_out; then BN1; then ReLU.
2. conv2 3x3, stride 1, padding 1, C_out -> C_out; then BN2 (no ReLU yet).
3. Shortcut: the identity x when stride == 1 and C_in == C_out; otherwise a projection: 1x1 conv with stride stride (C_in -> C_out, no padding) followed by its own BatchNorm.
4. Output: out = ReLU(main + shortcut). The ReLU comes after the addition.

The convs have no bias (BatchNorm's beta makes it redundant).

Inputs

• x: float array (N, C_in, H, W) (NCHW).
• params: dict from init_block_params(C_in, C_out, stride) with
  • 'conv1_w' (C_out, C_in, 3, 3), 'bn1_gamma', 'bn1_beta' (C_out,)
  • 'conv2_w' (C_out, C_out, 3, 3), 'bn2_gamma', 'bn2_beta' (C_out,)
  • only for a projection: 'proj_w' (C_out, C_in, 1, 1), 'proj_gamma', 'proj_beta' (C_out,)
• stride: 1 or 2, the stride of conv1 and of the projection.

Provided helpers (already in the starter code)

• conv2d(x, w, stride, padding): NCHW cross-correlation, w of shape (C_out, C_in, k, k), no bias.
• batch_norm2d(x, gamma, beta): training-mode BatchNorm: per-channel batch statistics over (N, H, W), biased variance, eps = 1e-5, like nn.BatchNorm2d in .train() mode.
• init_block_params(c_in, c_out, stride): deterministic parameters (adds proj_* only when a projection is needed).

Expected return format

A float array of shape (N, C_out, H_out, W_out) with H_out = (H - 1) // stride + 1 (that is (H + 2*1 - 3) // stride + 1), same for W. Every value is >= 0.

Edge cases and conventions

• Decide identity vs projection from the shapes: C_out = params['conv1_w'].shape[0].
• Both the main path and the shortcut must downsample by the same stride, or the addition fails.

Constraints

Use the provided helpers; NumPy only.

Follow-up

• How does the bottleneck block of ResNet-50 (1x1 reduce, 3x3, 1x1 expand) cut compute?
• What does the pre-activation ResNet (BN-ReLU-conv, no ReLU after the addition) change, and why does it help very deep nets?
• At inference BatchNorm uses running statistics. How would you fold it into the preceding conv?

Input: x: (2, 4, 6, 6), params = init_block_params(4, 4, 1), stride = 1
Output: shape (2, 4, 6, 6)

Input: x: (2, 3, 6, 6), params = init_block_params(3, 8, 2), stride = 2
Output: shape (2, 8, 3, 3)

Input: params with conv1_w and conv2_w set to zeros, stride = 1
Output: ReLU(x + bn2_beta)

import numpy as np

def conv2d(x, w, stride=1, padding=0):
    """Convolution (cross-correlation), no bias. x: (N, C_in, H, W), w: (C_out, C_in, k, k)."""
    N, C, H, W = x.shape
    k = w.shape[2]
    xp = np.pad(x, ((0, 0), (0, 0), (padding, padding), (padding, padding)))
    H_out = (H + 2 * padding - k) // stride + 1
    W_out = (W + 2 * padding - k) // stride + 1
    out = np.zeros((N, w.shape[0], H_out, W_out))
    for a in range(k):
        for b in range(k):
            patch = xp[:, :, a:a + stride * H_out:stride, b:b + stride * W_out:stride]
            out += np.einsum('nchw,oc->nohw', patch, w[:, :, a, b])
    return out


def batch_norm2d(x, gamma, beta, eps=1e-5):
    """Training-mode BatchNorm2d: normalize each channel with batch statistics over (N, H, W)."""
    mu = x.mean(axis=(0, 2, 3), keepdims=True)
    var = x.var(axis=(0, 2, 3), keepdims=True)
    return gamma[None, :, None, None] * (x - mu) / np.sqrt(var + eps) + beta[None, :, None, None]


def init_block_params(c_in, c_out, stride=1):
    """Deterministic parameters for a basic block (proj_* only if a projection is needed)."""
    def weight(shape, phase):
        fan_in = shape[1] * shape[2] * shape[3]
        return np.sin(np.arange(np.prod(shape)) * 1.3 + phase).reshape(shape) / np.sqrt(fan_in)

    def affine(c, phase):
        return 1 + 0.1 * np.cos(np.arange(c) + phase), 0.05 * np.sin(np.arange(c) + phase)

    p = {'conv1_w': weight((c_out, c_in, 3, 3), 0.0), 'conv2_w': weight((c_out, c_out, 3, 3), 1.0)}
    p['bn1_gamma'], p['bn1_beta'] = affine(c_out, 0.0)
    p['bn2_gamma'], p['bn2_beta'] = affine(c_out, 2.0)
    if stride != 1 or c_in != c_out:
        p['proj_w'] = weight((c_out, c_in, 1, 1), 2.0)
        p['proj_gamma'], p['proj_beta'] = affine(c_out, 4.0)
    return p


def residual_block(x, params, stride=1):
    """
    Basic residual block (training-mode BatchNorm).

    main     = BN2(conv2(ReLU(BN1(conv1(x)))))      conv1: 3x3, stride, pad 1; conv2: 3x3, pad 1
    shortcut = x                                   if stride == 1 and C_in == C_out
             = BN_proj(conv1x1(x, stride))         otherwise
    out      = ReLU(main + shortcut)

    Args:
        x: Input, shape (N, C_in, H, W)
        params: dict from init_block_params(C_in, C_out, stride)
        stride: Stride of conv1 (and of the projection)

    Returns:
        Output, shape (N, C_out, H_out, W_out) with H_out = (H - 1) // stride + 1
    """
    # Your code here
    pass