5.5 Stacking Units is Learning \(\phi\)

5.5.1 Hidden Units

Recall the house prices example, and take one house with 4 features, e.g.  \[x=(\text{size},\;\#\text{beds},\;\text{zip},\;\text{wealth})\in\mathbb{R}^4\]

In this scenario, imagine three hidden quantities

\[ \begin{aligned} a_1&=\mathrm{ReLU}(\beta_1 x_1+\beta_2 x_2+\beta_3) &&\text{"family size"},\\ a_2&=\mathrm{ReLU}(\beta_4 x_3+\beta_5) &&\text{"walkable"},\\ a_3&=\mathrm{ReLU}(\beta_6 x_3+\beta_7 x_4+\beta_8) &&\text{"school quality"}, \end{aligned} \]

and a linear \(\bar f_\beta(x)=\beta_9 a_1+\beta_{10}a_2+\beta_{11}a_3+\beta_{12}\).

The labels “family size” and “school quality” are only nicknames to make the picture readable. Fitting the net does not force \(a_3\) to mean school quality, and you should not read the hidden units that way.

What is forced is that the last-layer is still a linear model: once \(a=(a_1,a_2,a_3)\) is in hand, \(\hat y\) is ordinary linear regression of price on \(a\). The new piece is that \(a\) is not a column in the “design matrix”. It is computed from the raw \(x\), and the weights inside that computation are part of \(\beta\).

 

5.5.2 Fully Connected Layers

In fully connected layers, we stop choosing which inputs feed the hidden unit. A fully connected layer does the opposite: every hidden unit is allowed to use every coordinate of \(x\in\mathbb{R}^d\), and training decides which connections matter.

For \(j=1,\ldots,m\),

\[ z_j=(w_j^{[1]})^\top x+b_j^{[1]}, \qquad a_j=\mathrm{ReLU}(z_j), \qquad \bar f_\beta(x)=(w^{[2]})^\top a+b^{[2]}. \]

Superscripts \([1]\) and \([2]\) label layers of parameters.

 

Stack the first-layer weights as rows of \(W^{[1]}\in\mathbb{R}^{m\times d}\). Then \[ a=\mathrm{ReLU}\bigl(W^{[1]}x+b^{[1]}\bigr), \qquad \bar f_\beta(x)=W^{[2]}a+b^{[2]}. \]

That is a one-hidden-layer net. A deeper net iterates the same pattern: set \(a^{[0]}:=x\) and

\[ a^{[k]}=\mathrm{ReLU}\bigl(W^{[k]}a^{[k-1]}+b^{[k]}\bigr), \qquad k=1,\ldots,r-1, \]

\[ \bar f_\beta(x)=W^{[r]}a^{[r-1]}+b^{[r]}. \]

If \(a^{[k]}\in\mathbb{R}^{m_k}\) then \(W^{[k]}\in\mathbb{R}^{m_k\times m_{k-1}}\) and \(b^{[k]}\in\mathbb{R}^{m_k}\), with \(m_0=d\). For scalar regression, \(m_r=1\). The parameter count, including biases, is

\[ (d+1)m_1+(m_1+1)m_2+\cdots+(m_{r-1}+1)m_r. \]

The name multilayer perceptron is historical, since these are not Rosenblatt perceptrons. They are the previous chapter’s linear model on top of a \(\phi_c\) that is itself an iterated affine-plus-\(\sigma\) map.

If we write \(c\) for every parameter except the last layer, then \(a^{[r-1]}=\phi_c(x)\) and

\[ \bar f_\beta(x)=W^{[r]}\,\phi_\beta(x)+b^{[r]}. \]

  • Freeze \(c\): nonlinear regression with a fixed feature map.

  • Train \(c\): the feature map is part of the estimate.

The statistical cost of the second option is that \(\phi_c\) can interpolate noise. The statistical gain is that \(\phi_c\) can discover a corner or an interaction that we omitted.