Linear Regression with Gradient Descent - Supervised Learning

Linear Regression with Gradient Descent

Fit a one-feature linear model y ≈ w * x + b by running batch gradient descent on the mean squared error.

Background

Linear regression has a closed-form solution, but gradient descent is how almost every other model (and every neural network) is trained. Implementing it on the simplest model shows the full loop: predict, measure the loss, compute gradients, step.

Formula

Loss (mean squared error over the n samples):

J(w, b) = (1/n) * sum_i (y_pred_i - y_i)^2,     y_pred_i = w * x_i + b

Gradients:

dJ/dw = (2/n) * sum_i (y_pred_i - y_i) * x_i
dJ/db = (2/n) * sum_i (y_pred_i - y_i)

Algorithm:

1. Start from w = 0, b = 0.
2. Repeat iterations times: compute y_pred with the current w, b, compute both gradients, then update both parameters simultaneously: w -= learning_rate * dJ/dw, b -= learning_rate * dJ/db.
3. Return the final (w, b).

Inputs

• X: 1D float array of inputs, shape (n,)
• y: 1D float array of targets, shape (n,)
• learning_rate: step size (float, e.g. 0.01 to 0.1)
• iterations: number of full-batch updates (int)

Expected Return Format

A tuple (w, b) with both values rounded to 2 decimals.

Edge Cases and Conventions

• Use the exact update order above: the tests include a 10-iteration run, which checks the update rule itself, not just the optimum it converges to.
• Too large a learning rate makes the iterates diverge; the tests use stable values.

Constraints

• NumPy only; no np.polyfit or closed-form least squares. Vectorise each step over the samples (one Python loop over iterations is expected).

Follow-up

• Derive the closed form w = cov(x, y) / var(x), b = mean(y) - w * mean(x), and explain when gradient descent is still preferable.
• Why does standardising x let you use a larger learning rate?

Input: linear_regression(np.array([1, 2, 3, 4]), np.array([2, 4, 6, 8]), 0.1, 1000)
Output: (2.0, 0.0)

Input: linear_regression(np.array([1, 2, 3, 4]), np.array([2, 4, 6, 8]), 0.01, 1)
Output: (0.3, 0.1)

import numpy as np

def linear_regression(X: np.ndarray, y: np.ndarray,
                      learning_rate: float = 0.01,
                      iterations: int = 1000) -> tuple:
    """
    Train y = w * x + b with batch gradient descent on the MSE.

    Args:
        X: Input features, 1D array of shape (n,)
        y: Target values, 1D array of shape (n,)
        learning_rate: Step size for gradient descent
        iterations: Number of full-batch updates

    Returns:
        Tuple (w, b), each rounded to 2 decimals
    """
    # Initialize parameters
    w = 0.0
    b = 0.0
    n = len(X)

    # Your gradient descent implementation here
    pass

    return (round(w, 2), round(b, 2))