Logistic Regression with Gradient Descent - Supervised Learning

Logistic Regression with Gradient Descent

Train a binary logistic regression classifier from scratch: a linear score, a sigmoid, and batch gradient descent on the mean binary cross-entropy.

Background

Logistic regression is the baseline classifier in industry (click-through prediction, fraud scoring) and the "one-neuron network" of deep learning. It combines the two earlier problems in this section: Sigmoid for Logistic Regression and Binary Cross-Entropy Loss. The neat fact interviewers ask about: the gradient of BCE through a sigmoid simplifies to (prediction - label) * input, exactly the same form as linear regression.

Formula

For X of shape (m, n), w of shape (n,) and a scalar b:

z      = X @ w + b                      # logits, shape (m,)
y_pred = sigmoid(z) = 1 / (1 + exp(-z)) # probabilities, shape (m,)
L      = -(1/m) * sum_i [ y_i log(y_pred_i) + (1 - y_i) log(1 - y_pred_i) ]

dL/dw  = (1/m) * X.T @ (y_pred - y)     # shape (n,)
dL/db  = (1/m) * sum_i (y_pred_i - y_i) # scalar

Algorithm:

1. Start from w = zeros(n), b = 0.
2. Repeat iterations times: forward pass, both gradients, then w -= learning_rate * dw and b -= learning_rate * db.

Inputs

• X: features, array of shape (m, n) (ints or floats)
• y: binary labels in {0, 1}, shape (m,)
• learning_rate: step size (float)
• iterations: number of full-batch updates (int)

A sigmoid(z) helper is already provided in the starter code.

Expected Return Format

A tuple (w, b):

• w: NumPy array of shape (n,), rounded to 4 decimals (np.round(w, 4))
• b: Python float, rounded to 4 decimals

Edge Cases and Conventions

• There is no regularisation, so on linearly separable data (like the AND gate in the tests) the weights keep growing the longer you train; the tests fix the iteration count so the result is deterministic.
• You do not need to compute the loss itself to train; only its gradient.

Constraints

• NumPy only, vectorised over samples; no sklearn. (This matches sklearn.linear_model.LogisticRegression(penalty=None) only in the limit of convergence; sklearn adds an L2 penalty by default.)

Follow-up

• Derive dL/dz = y_pred - y from the chain rule.
• Add L2 regularisation (lam/2) * ||w||^2: how do the gradients change, and why is b usually not penalised?

Input: logistic_regression(np.array([[0, 0], [0, 1], [1, 0], [1, 1]]), np.array([0, 0, 0, 1]), 0.5, 1000)
Output: (array([6.0141, 6.0141]), -9.1984)

Input: Same data, iterations=1
Output: (array([0., 0.]), -0.125)

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def logistic_regression(X, y, learning_rate=0.1, iterations=1000):
    """
    Train logistic regression with batch gradient descent on the mean BCE.

    Args:
        X: Features, shape (m, n)
        y: Binary labels in {0, 1}, shape (m,)
        learning_rate: Step size
        iterations: Number of full-batch updates

    Returns:
        w: Learned weights, shape (n,), rounded to 4 decimals
        b: Learned bias (Python float), rounded to 4 decimals
    """
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0

    # Your code here
    pass

    return np.round(w, 4), round(float(b), 4)