NumPy Fundamentals

NumPy Fundamentals

Why it matters

Every ML system speaks arrays. A batch of images is a (batch, height, width, channels) array, a linear layer's weights are a (d_in, d_out) matrix, and PyTorch tensors copy NumPy's API almost name for name. Interviewers use NumPy questions to check that you think in whole arrays instead of Python loops, and most "the model trains but the numbers are wrong" bugs turn out to be shape or axis mistakes.

Core ideas

An array is a buffer plus a shape

An ndarray is one flat block of memory plus metadata: a dtype (float64, int64, ...), a shape and strides (how many bytes to jump per axis). Many operations only rewrite the metadata, so they are O(1) and return a view that shares memory with the original:

Operation	Returns	Cost
Basic slice a[1:3, ::2]	view	O(1)
reshape (contiguous input), .T, transpose	view	O(1)
Fancy index a[[0, 2]], boolean mask a[a > 0]	copy	O(k)
flatten(), arithmetic, reductions	new array	O(n)

Creation

np.zeros(shape), np.ones(shape) and np.eye(n) take sizes; np.arange(stop) (half-open, excludes stop) and np.linspace(start, stop, num) (closed, includes stop) take ranges.

Axes and reductions

arr.sum(axis=k) removes axis k. For a (n_rows, n_cols) array, axis=0 gives one value per column and axis=1 one value per row. keepdims=True keeps the reduced axis with size 1, which is what you want when you subtract the result back from the array.

Broadcasting

Two shapes are compatible when, compared from the right, each pair of dimensions is equal or one of them is 1 (missing leading dimensions count as 1). Size-1 dimensions are stretched without copying:

Left	Right	Result
(batch, d)	(d,)	(batch, d): a bias added to every row
(batch, d)	(batch, 1)	(batch, d): a per-row scale
(n, 1)	(1, m)	(n, m): an outer operation
(3, 4)	(3,)	error: 4 and 3 don't match

Reshape versus transpose

Arrays are stored in row-major (C) order, so the last axis changes fastest. reshape keeps that order and regroups it: entry [i, j] of a (4, 6) reshape is element 6*i + j of the vector. transpose keeps the data and reorders the axes: a.T[j, i] = a[i, j]. Both can produce a (6, 4) array from a (4, 6) one, with different values.

Worked example: standardising features

Standardisation turns each column into mean 0 and standard deviation 1, using one reduction and one broadcast:

X = np.array([[1., 10.],
              [2., 20.],
              [3., 30.]])          # (n_samples=3, n_features=2)
mu = X.mean(axis=0)                # [ 2., 20.]        shape (2,)
sigma = X.std(axis=0)              # [0.8165, 8.165]   shape (2,), population std
Z = (X - mu) / sigma               # (3, 2) - (2,) broadcasts over rows
# Z = [[-1.2247, -1.2247],
#      [ 0.    ,  0.    ],
#      [ 1.2247,  1.2247]]

Both columns end up identical because the second column is just 10 times the first: standardisation removes scale.

Common pitfalls

• (n,) versus (n, 1): a 1D array lines up with the last axis. X - X.mean(axis=1) fails for non-square X and, worse, silently subtracts along the wrong axis when X is square. Use keepdims=True.
• Views alias memory: b = a[0]; b[:] = 0 zeroes a row of a. Call .copy() when you need independence.
• Integer dtypes truncate: assigning 0.7 into an int64 array stores 0. Create float arrays when you'll store floats.
• ddof: np.std divides by N (ddof=0); torch.std and pandas divide by N - 1 by default.
• Float equality: compare with np.allclose, never ==.
• Accidental huge broadcasts: (n, 1, d) - (1, m, d) builds an (n, m, d) array. That's fine for pairwise distances on small data, and an out-of-memory error at scale.

Interview angle

• "Vectorise this loop" is the classic opener: replace per-element Python with one array expression, then explain why it's faster (no interpreter overhead per element, contiguous memory, SIMD).
• Expect "what shape comes out?" questions about broadcasting, axis= and keepdims, and be ready to state the broadcasting rule from memory.
• Complexity: elementwise ops and reductions are O(n); reshape, transpose and slicing are O(1) views; fancy indexing copies.
• Follow-ups: views versus copies, contiguity (why PyTorch's .view fails after a transpose), and the memory cost of broadcasting.

Problems in this section

1. Array Creation Methods: build zeros, ones, arange, linspace and identity arrays and learn which take shapes and which take ranges.
2. Indexing and Slicing: extract rows, columns, sub-blocks, the diagonal and reversed rows, with half-open slices.
3. Broadcasting Operations: centre rows and columns, add a bias and scale rows without loops.
4. Aggregation Functions: global, per-row and per-column statistics plus a 2D argmax/argmin.
5. Reshape and Transpose: regroup a vector into 2D and 3D arrays, permute axes and flatten back.

Next: Python Basics for ML for quick vectorisation warm-ups, Einstein Summation for a single notation covering all of these operations, and Normalize Features in Data Preprocessing, which reuses the standardisation above.