5.3 Illustration: Predicting House Prices
Our goal is to predict the selling price \(y\in\mathbb{R}\) of a house, in thousands of dollars, from a given set of features: size, bedrooms, zip code, and a neighbourhood wealth index. A snapshot of the data is:
| Size (ft\(^2\)) | Beds | ZIP | Wealth | Price ($1000) |
|---|---|---|---|---|
| 850 | 1 | 94305 | 0.4 | 420 |
| 1200 | 2 | 94305 | 0.6 | 610 |
| 2400 | 4 | 94301 | 1.2 | 980 |
| 3100 | 4 | 94022 | 2.1 | 1640 |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
Each row corresponds to an observation \((x_{i},y_{i})\), i.e. a house.
A linear predictor \(f(x)=\beta^\top x\) that uses size has two problems: Nothing stops \(\hat y\) from going negative for a small house, and a single slope cannot encode a floor such as “below \(1100\) ft², extra square footage does not change the price”.
Last chapter’s fix in such a problem was to put that floor into \(\phi\) by hand: add the column \(\mathrm{ReLU}(\mathrm{size}-1100)\), for example, and run OLS on the expanded features. This chapter’s fix is to leave the columns raw and let hidden units grow that floor .
It is worth trying to do the following calculation on one toy example:
Take \((x,y)=(2,3)\), \(w=1\), \(b=-1\).
Then \(z=wx+b=1 \, \Rightarrow \, a=\mathrm{ReLU}(z)=1\), and \(J=\tfrac12(1-3)^2=2\).
Since \(z>0\), \(\mathrm{ReLU}'(z)=1\), so \[ \frac{\partial J}{\partial a}=-2, \qquad \frac{\partial J}{\partial z}=-2, \qquad \frac{\partial J}{\partial w}=-4, \qquad \frac{\partial J}{\partial b}=-2. \]
A step of size \(\alpha=0.1\) yields \(w\leftarrow 1.4\) and \(b\leftarrow -0.8\). The next \(a\) moves toward \(3\). That is backpropagation on one scalar unit.
The simulated house prices can be downloaded here houses.csv. A detailed analysis of the house prices data can be found here: R and Python.