1.3 Why Statistical Learning is Challenging

In the framework of the previous section we reduced the learning problem to the estimation of a parameter vector \(w\), under the assumption that the true function \(f\) is known up to this parameter. In practice, however, we rarely know the exact form of \(f\). We must therefore approximate it by some function \(\hat{f}\). This approximation introduces an additional source of error.

Moreover, the objective function we minimize is computed on a training sample. While this quantifies how well the model fits the observed data, the real goal of statistical learning is good performance on new, unseen data. In other words, we aim to minimize the test error (also called generalization error), not the training error.

Statistical learning is therefore challenging for several reasons:

  • The training error typically underestimates the test/generalization error.

  • A model may perform well on the training data yet poorly on future data because of overfitting. The number of parameters required to approximate \(f\) can be large—sometimes larger than the number of available observations.

  • As the number of model parameters grows, the test error often increases substantially.

 

Two simple illustrations make these points concrete:

  1. Classification: 1-Nearest Neighbour
    The one-nearest-neighbour classifier achieves zero training error (it predicts every training point perfectly) but typically performs poorly on test data. An instructive demonstration of how increasing dimensionality affects the performance of linear classifiers can be found here.

  2. Regression: the \(p\) versus \(n\) problem
    Consider the linear model \[ \mathbf{y}_{n\times 1} = \mathbf{X}_{n\times p}\mathbf{w}_{p\times 1} + \varepsilon, \] where \(\mathbf{y}\) is the response vector, \(\mathbf{X}\) is the design matrix with \(p\) predictors, and \(\mathbf{w}\) contains the coefficients to be estimated.

    • When \(p < n\) we have more equations than parameters; classical regression methods usually work well.
    • When \(p = n\) we have exactly as many equations as parameters and can achieve a perfect fit on the training data.
    • When \(p > n\) we have fewer equations than parameters. In this high-dimensional regime classical methods break down and specialized techniques are required.

 

In summary, statistical learning is challenging because we must approximate an unknown function \(f\), control the gap between training and test error, and often operate in regimes where the number of parameters is large relative to the sample size. Successfully addressing these issues is essential for building models that generalize well to new data.