5.8 Capacity, Nonconvexity, and Penalty

In the previous chapter \(J(\beta)\) for OLS or ridge is convex. The \(J(\beta)\) of even a shallow net is not! Hidden units can be permuted, signs can flip when \(\sigma\) is odd, and entire regions of parameter space implement essentially the same function. SGD from two seeds can stop at two different predictors with similar training loss and different test loss. The practical response is the one already used for bandwidths and penalties: several starts, a holdout, and a simple regularizer.

Width and depth are capacity. A one-hidden-layer net with \(m\) units can form up to \(m\) corners in one dimension and can interpolate \(n\) points once \(m\) is large enough relative to \(n\). Interpolation is not automatically bad, but it is a reason to look at test MSE as \(m\) grows, not only at training \(J\).

The regularizer that sits closest to last chapter is weight decay. Adding \(\lambda\sum_k\|W^{[k]}\|_F^2\) to \(J\) is ridge on the weights. In nnet this is the decay argument and it pulls hidden units toward zero.

This does not replace a test set!