Chapter 3 Variable Selection & Regularization
Consider the multiple linear regression model with \(p\) predictors plus an intercept: \[Y \sim 1+ X_1+X_2+ \ldots+X_p\]
In many applications the number of explanatory variables \(p\) is large; in some cases we even have \(p \gg n\). This does not mean that every variable is relevant to the response \(Y\). In fact, only a small fraction of the \(p\) variables is typically believed to be relevant.
Our goal in this chapter is to develop methods that efficiently identify the predictors that are useful for estimating or predicting the response. Because the least-squares estimator \(\hat{\beta}\) is unbiased, the coefficients of irrelevant variables tend to be close to zero. Nevertheless, if our primary objective is good prediction, we must still ask whether it is beneficial to remove those unnecessary variables from the model.
To understand the consequences of including superfluous predictors, we first examine the training and testing errors more carefully.