Gaussian Naive Bayes - Supervised Learning

Gaussian Naive Bayes

Implement gaussian_nb_fit and gaussian_nb_predict_proba: fit one Gaussian per class and feature, then turn new points into class probabilities with Bayes' rule, computed in log space.

Background

Naive Bayes assumes the features are independent given the class, so the class-conditional density factorises into one 1D density per feature. The assumption is rarely true, yet the classifier is fast, needs little data and is a classic text and tabular baseline. The interview content is in the numerics: multiplying many small densities underflows, so everything happens in log space with a logsumexp.

Fit

For each class c in ascending order (np.unique(y)), with n_c samples of class c out of n:

prior[c]   = n_c / n
mean[c, j] = mean of feature j over the samples of class c
var[c, j]  = variance (ddof=0) of feature j over class c  +  epsilon
epsilon    = var_smoothing * max_j Var(X[:, j])     # computed once, over ALL samples

epsilon keeps variances away from zero (a feature that is constant within a class would otherwise divide by zero). These are exactly the conventions of scikit-learn's GaussianNB(var_smoothing=1e-9): priors = class_prior_, means = theta_, vars = var_.

Predict

The joint log-likelihood of sample x (shape (d,)) and class c:

jll[c] = log prior[c] - 0.5 * sum_j log(2*pi*var[c, j])
                      - 0.5 * sum_j (x_j - mean[c, j])^2 / var[c, j]

Normalise with logsumexp to get the posterior:

log P(c | x) = jll[c] - logsumexp_c'(jll[c'])
logsumexp(a) = max(a) + log(sum(exp(a - max(a))))

Inputs

• X: float array (n, d) for fitting, (m, d) for prediction
• y: class labels (n,)
• var_smoothing: float, default 1e-9

Expected Return Format

gaussian_nb_fit returns a dictionary with these keys:

• 'classes': sorted unique labels, shape (C,)
• 'priors': class priors, shape (C,)
• 'means': per-class feature means, shape (C, d)
• 'vars': per-class feature variances including epsilon, shape (C, d)

gaussian_nb_predict_proba returns an array of shape (m, C) whose rows sum to 1, with columns in the order of model['classes'].

Edge Cases and Conventions

• Use the population variance (ddof=0), as sklearn does.
• Never multiply raw densities: with many features, or a point far from every class, they underflow to 0 and you get 0/0 = nan. In log space a point at x = -500 still gets a clean [1.0, 0.0].

Constraints

• NumPy only; no sklearn or scipy. Vectorise prediction over samples and classes with broadcasting.

Follow-up

• Where does the "naive" assumption hurt most (duplicated or strongly correlated features)?
• How do multinomial and Bernoulli Naive Bayes differ, and why do they need Laplace smoothing instead of var_smoothing?

Input: gaussian_nb_fit(np.array([[1.0], [2.0], [3.0], [6.0], [7.0], [8.0]]), np.array([0, 0, 0, 1, 1, 1]))
Output: priors [0.5, 0.5], means [[2.0], [7.0]], vars ≈ [[0.6667], [0.6667]]

Input: gaussian_nb_predict_proba(model, np.array([[4.5]]))
Output: array([[0.5, 0.5]])

import numpy as np

def gaussian_nb_fit(X, y, var_smoothing=1e-9):
    """
    Fit Gaussian Naive Bayes.

    Args:
        X: Features, shape (n, d)
        y: Class labels, shape (n,)
        var_smoothing: Fraction of the largest feature variance added to all variances

    Returns:
        dict with 'classes' (C,), 'priors' (C,), 'means' (C, d),
        'vars' (C, d) (ddof=0 variances plus epsilon)
    """
    # Your code here
    pass


def gaussian_nb_predict_proba(model, X):
    """
    Posterior class probabilities, computed in log space.

    Args:
        model: dict returned by gaussian_nb_fit
        X: Features, shape (m, d)

    Returns:
        Probabilities, shape (m, C), columns in the order of model['classes'];
        rows sum to 1
    """
    # Your code here
    pass