3.1 Training vs. Testing Errors
Consider that we split our data in two parts:
Training data \(\{\mathbf{x}_i, y_i \}_{i=1}^n\): used to fit the model
Testing data \(\{\mathbf{x}_i, y_i^* \}_{i=1}^n\): an independent data set collected at the same locations \(\mathbf{x}_i\)
Remark: In practice we are given a single data set that we randomly partition into a training set and a testing set (typically allocating roughly 80 % of the observations to training).
We assume that both testing and training observations come from the same population (or are observed at the same design points \(\mathbf{x}_i\)). Statistically we may therefore write
\[\mathbf{y}_{n\times 1},\,\, \mathbf{y}^*_{n\times 1} \sim^{iid} N_n(\mathbf{\mu},\sigma^2 \mathbf{I}_n) \text{ and } \mathbf{\mu}=\mathbf{X}\mathbf{\beta}\]
or equivalently,
\[ \mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{\varepsilon},\, \mathbf{\varepsilon} \sim^{iid} \mathcal{N}_n(\mathbf{0},\sigma^2 \mathbf{I}_n) \quad \text{ and } \quad \mathbf{y}^* =\mathbf{X}\mathbf{\beta} + \mathbf{\varepsilon}^*,\, \mathbf{\varepsilon}^* \sim^{iid} \mathcal{N}_n(\mathbf{0},\sigma^2 \mathbf{I}_n) \] \[ \text{ with }\,\, \mathbf{\varepsilon}, \mathbf{\varepsilon}^* \,\, \text{ independent}.\]
Having these models in mind, we compute the MSE for train and testing data:
For simplicity we suppress the intercept in the calculations below (so the model dimension is \(p\) rather than \(p+1\)). The expected training and testing mean squared errors are then:
\[\begin{align*} {\color{blue} {\mathbb{E}\bigl(\text{Train Error}\bigr)}} & = \mathbb{E}\Bigl|\Bigl|\mathbf{y}-\hat{\mathbf{y}} \Bigr|\Bigr|^2=\mathbb{E}\Bigl|\Bigl|(\mathbf{I - H})\mathbf{y}\Bigr|\Bigr|^2\\ &=tr \Bigl((\mathbf{I - H})Cov(\mathbf{y})(\mathbf{I - H})^T \Bigr)\\ & = \sigma^2 tr\Bigl( (\mathbf{I - H}) \Bigr)= (n-p) \sigma^2\\ &= {\color{blue} {n\sigma^2 \mathbf{-} p \sigma^2}}\\ \end{align*}\]
\[\begin{align*} {\color{blue} {\mathbb{E}\bigl(\text{Test Error} \bigr)}} & = \mathbb{E}\Bigl|\Bigl|\mathbf{y}^*-\mathbf{X}\hat{\mathbf{\beta}}\Bigr|\Bigr|^2\\ &= \mathbb{E}\Bigl|\Bigl|(\mathbf{y}^*- \mathbf{X} \mathbf{\beta})+(\mathbf{X} \mathbf{\beta}-\mathbf{X}\hat{\mathbf{\beta}})\Bigr|\Bigr|^2\\ &= \mathbb{E}\Bigl|\Bigl|\mathbf{y}^* -\mathbf{\mu}\Bigr|\Bigr|^2 + \mathbb{E}\Bigl|\Bigl|\mathbf{X} \mathbf{\beta}-\mathbf{X}\hat{\mathbf{\beta}}\Bigr|\Bigr|^2\\ &= \mathbb{E}||\mathbf{\varepsilon}^*||^2 + tr\Bigl(\mathbf{X}Cov(\hat{\mathbf{\beta}})\mathbf{X}^T \Bigr)\\ &= n \sigma^2 +\sigma^2 tr\mathbf{H}\\ &= {\color{blue} {n \sigma^2 \mathbf{+} p \sigma^2}}\\ \end{align*}\]
From these expressions we see that
the training error decreases with \(p\) .
the testing error increases with \(p\) .
Consequently, if the sole objective is pure prediction, simply adding more columns to \(\mathbf{X}\) is not automatically beneficial. Does this imply that the intercept-only model (\(p=0\))—the model with the smallest expected test error under the calculation above—is optimal? No.
The preceding analysis rests on two important assumptions:
The conditional mean \(E(\mathbf{Y}|\mathbf{X})\) lies in the column space of \(\mathbf{X}\); that is, there exists a coefficient vector \(\beta\) such that \(\mathbb{E}(\mathbf{Y}\mid\mathbf{X}) =\mathbf{X}\beta\).
The design matrix \(\mathbf{X}\) contains all available predictors. When we fit a linear model that uses only a subset of the columns of \(\mathbf{X}\), an additional bias term appears.
More generally we may face a model of the form \[Y = f(X) + \varepsilon\] in which \(f\) is not linear.
Approximating \(f\) by a linear function \(f^*\) introduces bias. Even when the true relationship is linear, omitting relevant predictors likewise produces bias.
In the presence of bias the expected errors become:\[\begin{align*} \mathbb{E}\bigl(\text{Test Error}\bigr)^2 &= n\sigma^2 + \color{blue}{p\sigma^2} + \color{orange}{Bias}\\ \mathbb{E}\bigl(\text{Training Error}\bigr)^2 &= n\sigma^2 - \color{blue}{p\sigma^2} + \color{orange}{Bias} \end{align*}\] where \(p\sigma^2\) is the unavoidable error – the \(\varepsilon\) – and Bias is the model error.
Larger model (\(p\) large) \(\rightarrow\) small Bias, large Variance (\(p\sigma^2\)).
Smaller model (\(p\) small) \(\rightarrow\) large Bias, small Variance (\(p\sigma^2\)).
Reducing the test (prediction) error therefore requires finding a good trade-off between bias and variance.