5.1 Learning by Fitting a Model
A method uses data \(E\) to fit a model \(f\). We say it has learned if a performance measure \(P\) on a task \(T\) is better after seeing \(E\) than before:
\[ \text{data }E \longrightarrow \text{algorithm} \longrightarrow \text{model }f(\mathbf{x}). \]
Linear regression already satisfies this definition. Assume that the Task \(T\) is to predict a house’s price from size; the Measure \(P\) is the MSE, and the experience \(E\) are the past sales. Learning has occurred if the test MSE becomes smaller as more sales are observed.
A neural net is the same object, i.e. a predictor \(f\) fitted from \(E\), with a different parameterization of \(f\).
Consider a training set \(\{(x_{i},y_{i})\}_{i=1}^n\) with \(x_{i}\in\mathbb{R}^d\) and \(y_{i}\in\mathbb{R}\). Nonlinear regression means that
\[ f_\beta(x)=\beta^\top\phi(x) \] with criterion \[ J(\beta)=\frac1n\sum_{i=1}^n\frac12\bigl(f_\beta(x_{i})-y_{i}\bigr)^2, \]
and \(\beta\) is obtained by a linear method (OLS, ridge, a kernel solver). The map \(\phi\) might be monomials, splines, a radial basis, or an interaction.
Two important things to keep in mind:
The loss is a property of \(y\), not of \(\phi\). For a real-valued \(y\) we use a squared error.
Once \(\phi(x)\) is computed, the last step is ordinary linear regression of \(y\) on those features.
Every network below is built so that these remain true. If \(a=\phi_c(x)\) denotes the last hidden layer, the prediction is \(W a+b\):
Freeze \(c\) and you are back in the previous chapter.
Train \(c\) jointly with \((W,b)\) and you have a neural net.
The new challenges are statistical as much as algorithmic. The map \(\beta\mapsto J(\beta)\) is no longer convex, so two random starts can stop at two different predictors. Adding hidden units (width) or layers (depth) makes the training MSE easier to drive down: the model is larger, and a wide net can always mimic a narrow one.
If you think of the houses example, the same extra capacity can go either way. It helps if the extra units capture signal the smaller net missed (a corner, an interaction). It hurts if they fit noise instead.