5.5 Stacking Units is Learning \(\phi\)
5.5.2 Fully Connected Layers
In fully connected layers, we stop choosing which inputs feed the hidden unit. A fully connected layer does the opposite: every hidden unit is allowed to use every coordinate of \(x\in\mathbb{R}^d\), and training decides which connections matter.
For \(j=1,\ldots,m\),
\[ z_j=(w_j^{[1]})^\top x+b_j^{[1]}, \qquad a_j=\mathrm{ReLU}(z_j), \qquad \bar f_\beta(x)=(w^{[2]})^\top a+b^{[2]}. \]
Superscripts \([1]\) and \([2]\) label layers of parameters.
Stack the first-layer weights as rows of \(W^{[1]}\in\mathbb{R}^{m\times d}\). Then \[ a=\mathrm{ReLU}\bigl(W^{[1]}x+b^{[1]}\bigr), \qquad \bar f_\beta(x)=W^{[2]}a+b^{[2]}. \]
That is a one-hidden-layer net. A deeper net iterates the same pattern: set \(a^{[0]}:=x\) and
\[ a^{[k]}=\mathrm{ReLU}\bigl(W^{[k]}a^{[k-1]}+b^{[k]}\bigr), \qquad k=1,\ldots,r-1, \]
\[ \bar f_\beta(x)=W^{[r]}a^{[r-1]}+b^{[r]}. \]
If \(a^{[k]}\in\mathbb{R}^{m_k}\) then \(W^{[k]}\in\mathbb{R}^{m_k\times m_{k-1}}\) and \(b^{[k]}\in\mathbb{R}^{m_k}\), with \(m_0=d\). For scalar regression, \(m_r=1\). The parameter count, including biases, is
\[ (d+1)m_1+(m_1+1)m_2+\cdots+(m_{r-1}+1)m_r. \]
The name multilayer perceptron is historical, since these are not Rosenblatt perceptrons. They are the previous chapter’s linear model on top of a \(\phi_c\) that is itself an iterated affine-plus-\(\sigma\) map.
If we write \(c\) for every parameter except the last layer, then \(a^{[r-1]}=\phi_c(x)\) and
\[ \bar f_\beta(x)=W^{[r]}\,\phi_\beta(x)+b^{[r]}. \]
Freeze \(c\): nonlinear regression with a fixed feature map.
Train \(c\): the feature map is part of the estimate.
The statistical cost of the second option is that \(\phi_c\) can interpolate noise. The statistical gain is that \(\phi_c\) can discover a corner or an interaction that we omitted.