2.6 Rank deficiency

The design matrix \(\mathbf{X}\) is an \(n\times (p+1)\) matrix. If this matrix is not of full rank (i.e., its columns are not linearly independent), the matrix \(\mathbf{X}^T \mathbf{X}\) cannot be inverted. This means that there is redundancy among the columns of \(\mathbf{X}\): at least one column can be written as a linear combination of other columns. Consequently, \(\mathbf{X^T X}\) is singular and the LS solution is not unique (an identifiability problem).

Nevertheless, the column space \(\mathcal{C}(\mathbf{X})\) remains well-defined, so the fitted vector \(\hat{y}\) is still well-defined and can be computed. R and Python handle this situation, and the resulting model can still be used for prediction.