5.3 Illustration: Predicting House Prices

Our goal is to predict the selling price \(y\in\mathbb{R}\) of a house, in thousands of dollars, from a given set of features: size, bedrooms, zip code, and a neighbourhood wealth index. A snapshot of the data is:

Size (ft\(^2\)) Beds ZIP Wealth Price ($1000)
850 1 94305 0.4 420
1200 2 94305 0.6 610
2400 4 94301 1.2 980
3100 4 94022 2.1 1640
\(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\) \(\vdots\)

Each row corresponds to an observation \((x_{i},y_{i})\), i.e. a house.

 

A linear predictor \(f(x)=\beta^\top x\) that uses size has two problems: Nothing stops \(\hat y\) from going negative for a small house, and a single slope cannot encode a floor such as “below \(1100\) ft², extra square footage does not change the price”.

Last chapter’s fix in such a problem was to put that floor into \(\phi\) by hand: add the column \(\mathrm{ReLU}(\mathrm{size}-1100)\), for example, and run OLS on the expanded features. This chapter’s fix is to leave the columns raw and let hidden units grow that floor .

 

It is worth trying to do the following calculation on one toy example:
  • Take \((x,y)=(2,3)\), \(w=1\), \(b=-1\).

  • Then \(z=wx+b=1 \, \Rightarrow \, a=\mathrm{ReLU}(z)=1\), and \(J=\tfrac12(1-3)^2=2\).

  • Since \(z>0\), \(\mathrm{ReLU}'(z)=1\), so \[ \frac{\partial J}{\partial a}=-2, \qquad \frac{\partial J}{\partial z}=-2, \qquad \frac{\partial J}{\partial w}=-4, \qquad \frac{\partial J}{\partial b}=-2. \]

A step of size \(\alpha=0.1\) yields \(w\leftarrow 1.4\) and \(b\leftarrow -0.8\). The next \(a\) moves toward \(3\). That is backpropagation on one scalar unit.

   

The simulated house prices can be downloaded here houses.csv. A detailed analysis of the house prices data can be found here: R and Python.