7.2 Fisher Discriminant Analysis (FDA)
Supervised dimension reduction

FDA looks for directions \(a \in \mathbb{R}^p\) such that the projected data \(a^\top x\) are well separated by class: class means should be far apart relative to the spread within each class. Equivalently, we maximize the ratio \[ \frac{\text{between-class variation}}{\text{within-class variation}} = \frac{a^\top B a}{a^\top W a}. \] Here \(W\) and \(B\) are the within-class and between-class covariance matrices \[\begin{align*} W &= \frac{1}{n-K} \sum_{k=1}^{K} \sum_{i: y_i = k} (\mathbf{x}_i - \hat{\mu}_k)(\mathbf{x}_i - \hat{\mu}_k)^\top, \\ B &= \frac{1}{K-1} \sum_{k=1}^{K} n_k (\hat{\mu}_k - \bar{\hat{\mu}})(\hat{\mu}_k - \bar{\hat{\mu}})^\top, \end{align*}\] where \(\bar{\hat{\mu}} = n^{-1}\sum_i \mathbf{x}_i\) is the grand mean. The matrix \(W\) is exactly the pooled covariance \(\hat{\Sigma}\) from LDA. If we write each point as \[ \mathbf{x}_i = (\mathbf{x}_i - \hat{\mu}_{y_i}) + \hat{\mu}_{y_i}, \] then \(W\) is the covariance of the residuals \(\mathbf{x}_i - \hat{\mu}_{y_i}\), and \(B\) is (up to scaling) the covariance of the fitted class means \(\hat{\mu}_{y_i}\). Overall constants in \(B\) and \(W\) do not change the maximizer of the ratio.
Generalized Eigenvalue Problem
\[ \max_a \frac{a^\top B a}{a^\top W a} \quad\Longleftrightarrow\quad \max_a \; a^\top B a \quad \text{subject to} \quad a^\top W a = 1. \] Assume \(W = U D^2 U^\top\) and set \(b = W^{1/2} a\) with \(W^{1/2} = U D U^\top\), so \(W^{-1/2} = U D^{-1} U^\top\) and \(a = W^{-1/2} b\). Then \[\begin{align*} a^\top B a &= b^\top \bigl(W^{-1/2} B W^{-1/2}\bigr) b, \end{align*}\] subject to \(\|b\|^2 = 1\). This is an ordinary eigenvalue problem. Solve it sequentially:
- \(b_1 =\) leading eigenvector of \(W^{-1/2} B W^{-1/2}\), then \(a_1 = W^{-1/2} b_1\);
- \(b_2 =\) second eigenvector, then \(a_2 = W^{-1/2} b_2\);
- and so on.
The \(b_j\) are orthonormal, but the \(a_j\) are orthogonal only with respect to \(W\): \[\begin{align*} b_j^\top b_j &= 1, & a_j^\top W a_j &= 1, \\ b_j^\top b_\ell &= 0, & a_j^\top W a_\ell &= 0 \quad (j \ne \ell). \end{align*}\] At most \(K-1\) directions exist, because \(\operatorname{rank}(B) \le K-1\). Those directions span the same subspace obtained from reduced-rank LDA.
LDA vs. FDA
LDA is a classifier: it produces a decision rule. FDA is a supervised dimension-reduction method: it returns directions, not labels. FDA does not start from a Gaussian model, but the subspace it finds coincides with that of LDA when the class-conditional covariances are equal—the same working assumption that makes LDA’s boundary linear. Unlike PCA, which ignores \(Y\), FDA uses the labels. The same idea can be used as a pre-processing step for regression, but in high dimensions the estimated directions overfit unless we regularize.
Risk of overfitting

A caricature: \(n = 6\) points in high dimension, three from class 1 and three from class 0. One can always find a direction that sends the first class to (near) 1 and the second class to (near) 0. The separation is an artifact of \(p\) being comparable to \(n\); even pure noise features can be combined to separate the training labels.
Two simple remedies:
- Shrinkage / \(\ell_1\) penalties. Regularize the estimated means or the inverse covariances (or the FDA criterion itself) so that the estimated directions cannot use every coordinate.
- Independence. Pretend that the \(p\) coordinates are independent within each class, so that \(\Sigma_k\) is diagonal. That is naive Bayes, and it is often a surprisingly strong high-dimensional baseline.