7.3 Naive Bayes Classifiers
The multi-class Bayes rule is still \[ \arg\max_k P(Y = k \mid X = \mathbf{x}) = \arg\max_k \; \pi_k \, f_k(\mathbf{x}). \] Naive Bayes imposes the product factorization \[ f_k(\mathbf{x}) = \prod_{j=1}^{p} f_{kj}(x_j), \] i.e., the coordinates of \(\mathbf{x}\) are treated as independent given the class. Independence is almost never literally true; it is a regularizer for high-dimensional problems.
Each univariate density \(f_{kj}\) is estimated separately within class \(k\): histograms or multinomials for discrete features; kernel density estimates (nonparametric naive Bayes) or univariate Gaussians (parametric / Gaussian naive Bayes) for numeric features.
Summary of parametric (Gaussian) naive Bayes
Training. From \(\{(x_i, y_i)\}_{i=1}^{n}\), compute \(\hat{\pi}_k\) and, for each class \(k\) and coordinate \(j\), the marginal mean and variance \((\hat{\mu}_{kj}, \hat{\sigma}_{kj}^2)\).
Prediction at \(\mathbf{x}^* = (x_1^*, \ldots, x_p^*)\): \[\begin{align*} d_k(\mathbf{x}^*) &= -2\log \hat{\pi}_k + \sum_{j=1}^{p} \Biggl[ \log \hat{\sigma}_{kj}^2 + \frac{(x_j^* - \hat{\mu}_{kj})^2}{\hat{\sigma}_{kj}^2} \Biggr]. \end{align*}\] Predict \(\arg\min_k d_k(\mathbf{x}^*)\). (Additive constants that do not depend on \(k\) have been dropped.)
Summary
Discriminant analysis estimates the class-conditional factorization \[ P(X = x, Y = k) = P(X = x \mid Y = k)\, P(Y = k) \] and then recovers \(P(Y = k \mid X = x)\) by Bayes’ theorem. The idea is conceptually clean and works well when \(p\) is modest and the Gaussian (or product) model is not badly wrong. In high dimensions, unrestricted QDA/LDA overfit; pooling, shrinking, reducing rank, or assuming coordinate-wise independence are the usual ways to buy stability.