5.4 Same Loss, New \(f_\beta\)
The training process is the same as before:
You pick a loss \(J(\beta)\) that is small when \(\hat y\) is close to \(y\).
You compute a gradient.
You take a step that decreases \(J\).
What changed is only how \(\hat y=f_\beta(x)\) is computed.
For a real-valued price the natural loss is still a squared error. For observation \(i\), \[J_{i}(\beta)=\frac12\bigl(f_\beta(x_{i})-y_{i}\bigr)^2.\] On the whole training set we average over all the data: \[J(\beta)=\frac1n\sum_{i=1}^n J^{(i)}(\beta).\]
The factor \(1/2\) is just a scaling to make the “2” from differentiation of the square disappear.
The factor \(1/n\) is a convention. Dividing by \(n\) does not move the \(\beta\) that minimizes \(J\). It does change, however, the size of \(\nabla J\), so a step size \(\alpha\) that worked for the sum will be too small for the average.
The formula for \(J\) is the ordinary least-squares loss. If you freeze the hidden layers and treat the last layer as OLS on features \(\phi(x)\), you are minimizing exactly this \(J\). A neural net minimizes the same \(J\); it just uses a more complicated \(f\beta\).
To minimize \(J\) we still move \(\beta\) in the direction of \(-\nabla J\). In other words, the solver is no longer “solves the normal equations”, but rather “follows \(-\nabla J\)”. Since we do not get a closed form, we have to follow the gradient steps.
Three common updates differ only in how many observations (houses in our example) enter the gradient:
- Full Gradient Descent uses every training point at every step: \[\beta\leftarrow\beta-\alpha\,\nabla_\beta J(\beta).\] The direction is exact (for this \(J\)), but each step costs a pass over all \(n\) houses.
- Stochastic Gradient Descent picks one index \(j\) at random and pretends \(J^{(j)}\) is the whole objective: \[\beta\leftarrow\beta-\alpha\,\nabla_\beta J^{(j)}(\beta).\] Each step is cheap and noisy. Under standard conditions the noise averages out and \(\beta\) still heads toward a stationary point.
- Mini-batch SGD is the compromise used in practice. Draw \(B\) indices and average their gradients: \[\beta\leftarrow\beta-\frac{\alpha}{B}\sum_{k=1}^B\nabla_\beta J^{(j_k)}(\beta).\] \(B=32\) or \(B=64\) is typical.
Remark:
None of these procedures inherits the convexity of ridge. In the previous chapter, \(J(\beta)=\|y-\Phi\beta\|^2+\lambda\|\beta\|^2\) had one valley. Here \(f_\beta\) is a composition of nonlinear maps, so \(J\) can have many valleys. Two random starts can stop at two different predictors with similar training loss and different test loss. That is not a reason to avoid gradient methods. It is a reason to try more than one starts, and decide the “winner” on a test set rather than on training \(J\) alone.