2.9 Collinearity

In practice we often encounter data sets in which many of the predictors are highly correlated with one another. In such cases the fitted values and the sampling variances of the regression coefficients can depend heavily on which particular predictors are included in the model. The resulting estimators become sensitive and unreliable.

Possible symptoms of collinearity include :

  1. high pairwise (sample) correlation among the predictors,

  2. a relatively large \(R^2\),

  3. a statistically significant overall \(F\) test even though non of the individual predictors is statistically significant.

Specific diagnostic tests and metrics exist for quantifying the severity of collinearity, but they lie beyond the scope of this course.

The simplest remedy is to remove one or more predictors from each highly correlated group. More advanced approaches include principal-component analysis (PCA) and regularization methods based on penalized least squares.