5.6 Activations, Depth, and Approximation

ReLU is the hidden-unit nonlinearity we use in this chapter. Most other names you will hear are variations on the same idea: take the weighted sum \(z=w^\top x+b\), then squash or bend it before passing it on.

Two older choices are the sigmoid \(1/(1+e^{-z})\) and \(\tanh\). Both flatten out as \(z\to\pm\infty\): the unit’s output barely changes, and the derivative is almost \(0\). Gradients then die as they travel back through the layer. That is what “saturate” means. Those functions are still useful when you want an output in \((0,1)\) or \((-1,1)\), that is a probability, or a gate that turns a signal up or down, but they are a poor default in the middle of a regression net.

Leaky ReLU is ReLU with a small slope on the negative side instead of a hard zero, so a unit that lands below \(0\) can still send a little gradient. Swish (also called SiLU), GELU, and softplus are smooth versions of ReLU: same general shape, rounded at the corner so the derivative is defined everywhere.

A gated linear unit (GLU) is a different object. It is not a function \(\sigma:\mathbb{R}\to\mathbb{R}\). It takes a vector \(f\), builds two linear maps of \(f\), sends one through a sigmoid-like gate \(g\), and multiplies the two pieces coordinatewise. Large language models use blocks of this form. You do not need it for a continuous response.

 

If \(\sigma\) is the Identity, Depth does nothing

If \(\sigma\) is the identity, then even with two layers

\[ \bar f_\beta(x)=W^{[2]}W^{[1]}x=\widetilde W\,x. \]

Composition of linear maps is linear in which case Depth adds nothing and \(\phi\) is the identity. So, we are back at OLS on raw \(x\). Nonlinearity is what makes a hidden layer more than a change of basis that could have been absorbed into \(\beta\).

Why one ReLU is not enough, and what “enough” means

One ReLU is one corner. A target with two corners/bends needs at least two hidden units. In one dimension, a one-hidden-layer ReLU net with \(m\) units is piecewise linear with at most \(m\) corners. Width adds pieces while depth composes pieces.

A standard universal-approximation result (Cybenko 1989; Hornik 1991) says that one hidden layer with a non-polynomial activation, and enough units, can approximate any continuous function on a compact set of \(\mathbb{R}^d\) to arbitrary uniform accuracy.

A polynomial net (last chapter’s bases) builds curves from powers of \(x\). A ReLU net builds curves from straight pieces joined at corners: on each region the map is linear, and the regions are cut out by the hyperplanes \(w^\top x+b=0\). So, ReLU networks have the same kind of density, with continuous piecewise-linear maps.

The theorem does not give a width, a rate, or a training algorithm. It does not say the SGD minimizer of \(J\) on \(n\) noisy \(y\)’s will be close to the true regression function. It plays the same role here that Weierstrass played for polynomials: we already know the model class is rich enough to approximate reasonable functions. Whether a given \(n\) and a given width estimate something useful is a question that needs to be investigated empirically.