To illustrate the ideas in this chapter, we will predict house prices from a set of features including size, beds, zip, and a wealth index. This is a simulated example for illustration purposes, and the data can be found and downloaded in Section 5.3.

 

The response \(y\) (house price) is obviously continuous, so we will fit linear and nonlinear regression models, this is the specified \(\phi\) from the previous chapters, and then we will employ a neural net to learn \(\phi_c\) using a squared error loss. In the end we will compare the results on the testing set.

 

Specifically, we will fit:

  1. OLS on the raw features, \(\phi(x)=x\);

  2. OLS on a hand-built \(\phi\) (a corner in size, and interaction \(\mathrm{beds}\times\mathrm{wealth}\));

  3. OLS on a cubic in size plus the other raw features;

  4. a one-hidden-layer net with a linear last layer, two widths, with weight decay.

We will report \(\frac1n\sum_i(\hat y_i-y_i)^2\) on a test set.

 

library(tidyverse)
library(nnet)
set.seed(598)

relu <- function(z) pmax(z, 0)
mse  <- function(y, yhat) mean((y - yhat)^2)
houses <- read.csv("houses.csv", header = TRUE) %>%
  mutate(zip = factor(zip))

zip must be a factor which means that it will enter the regression as a et of dummy variables.

 

Scatter Plots

We start by investigating the pairwise relationship of the response vs. size and wealth grouped by zip code.  

houses %>%
  ggplot(aes(size, price, colour = beds)) +
  geom_point(alpha = 0.55, size = 1.6) +
  geom_smooth(method = "loess", se = FALSE, colour = "black", linewidth = 0.7) +
  geom_vline(xintercept = 1100, linetype = 2, colour = "grey50") +
  scale_colour_viridis_c() +
  labs(
    title = "Price vs size: a floor, then a slope",
    subtitle = "Dashed line: 1100 ft², where the size effect starts in the hidden law",
    x = "size (ft²)", y = "price ($1000s)",
    colour = "beds"
  ) +
  theme_minimal(base_size = 12)

  We can see that the straight line does not have a very good fit. There is a corner/elbow on the lower left of the plot.  

houses %>%
  ggplot(aes(wealth, price)) +
  geom_point(alpha = 0.4) +
  geom_smooth(method = "lm", se = FALSE, colour = "steelblue") +
  facet_wrap(~zip) +
  labs(
    title = "Wealth vs price, by zip",
    x = "neighbourhood wealth (index)", y = "price ($1000s)"
  ) +
  theme_minimal(base_size = 12)

 

Create Training and Testing Data

We start by scaling the numeric columns in the training set. For nnet, we also need to scale \(y\) since the nnet package is unstable when the output lives on a \(10^2\)–\(10^3\) scale.

 

index <- sample(seq_len(nrow(houses)), size = floor(0.75 * nrow(houses)))
train <- houses[index, ]
test  <- houses[-index, ]

make_xy <- function(df, scaler = NULL) {
  mm <- model.matrix(price ~ size + beds + zip + wealth, data = df)[, -1]
  if (is.null(scaler)) {
    scaler <- list(
      x_center = colMeans(mm),
      x_scale  = pmax(apply(mm, 2, sd), 1e-8),
      y_center = mean(df$price),
      y_scale  = sd(df$price)
    )
  }
  X <- scale(mm, center = scaler$x_center, scale = scaler$x_scale)
  y_sc <- (df$price - scaler$y_center) / scaler$y_scale
  list(X = X, y = df$price, y_sc = y_sc, scaler = scaler)
}

unscale_y <- function(y_sc, scaler) scaler$y_center + scaler$y_scale * y_sc

tr <- make_xy(train)
te <- make_xy(test, scaler = tr$scaler)

 

We are now ready to run the different types of “regressions”. For each model, we keep the MSE on the testing set to compare in the end.

 

1. Raw Features

This is the usual OLS which we means that we specify \(f_\beta(x)=\beta^\top x\) on size, beds, zip dummies, and wealth.

fit_raw <- lm(price ~ size + beds + zip + wealth, data = train)
pred_raw_tr <- predict(fit_raw, newdata = train)
pred_raw_te <- predict(fit_raw, newdata = test)
mse_raw <- c(train = mse(train$price, pred_raw_tr),
             test  = mse(test$price,  pred_raw_te))
coef(fit_raw)
##  (Intercept)         size         beds     zip94301     zip94305     zip95112 
## -261.6230314    0.2630489   51.3892150   -3.5935366   45.3503352    6.5097652 
##       wealth 
##  199.6762570
mse_raw
##    train     test 
## 2212.909 2622.138

 

2. Hand-built \(\phi\)

In the simulated data, we have a a floor on size at \(1100\) ft\(^2\), and \(\mathrm{beds}\times\mathrm{wealth}\). So, we fit an OLS that matches this specific behavior:

train_phi <- train %>%
  mutate(
    size_corner = relu(size - 1100),
    beds_wealth = beds * wealth
  )
test_phi <- test %>%
  mutate(
    size_corner = relu(size - 1100),
    beds_wealth = beds * wealth
  )

fit_phi <- lm(
  price ~ size_corner + beds + zip + wealth + beds_wealth,
  data = train_phi
)
pred_phi_tr <- predict(fit_phi, newdata = train_phi)
pred_phi_te <- predict(fit_phi, newdata = test_phi)
mse_phi <- c(train = mse(train$price, pred_phi_tr),
             test  = mse(test$price,  pred_phi_te))
coef(fit_phi)
## (Intercept) size_corner        beds    zip94301    zip94305    zip95112 
##   66.644408    0.281340   26.203743   -3.322533   49.660671   10.872722 
##      wealth beds_wealth 
##  118.070683   26.133155
mse_phi
##    train     test 
## 1378.671 1502.382

 

3. Cubic in size; Raw in other Features

 

fit_poly <- lm(price ~ poly(size, 3) + beds + zip + wealth, data = train)
pred_poly_tr <- predict(fit_poly, newdata = train)
pred_poly_te <- predict(fit_poly, newdata = test)
mse_poly <- c(train = mse(train$price, pred_poly_tr),
              test  = mse(test$price,  pred_poly_te))
mse_poly
##    train     test 
## 1547.147 1676.755

 

4. One-Hidden-Layer Net

nnet fits \[ a=\sigma(W^{[1]}x+b^{[1]}), \qquad \hat y=W^{[2]}a+b^{[2]}, \] with a sigmoid \(\sigma\) (not ReLU) and linout = TRUE. decay is weight decay. The net sees only the scaled raw design, not size_corner or interaction beds_wealth.

fit_one_nnet <- function(size = 8, decay = 0.01, maxit = 800) {
  nnet(
    x = tr$X, y = tr$y_sc,
    size = size, linout = TRUE, trace = FALSE,
    decay = decay, maxit = maxit, MaxNWts = 5000
  )
}

pick_best <- function(candidates) {
  candidates[[which.min(vapply(candidates, `[[`, numeric(1), "value"))]]
}

set.seed(598)
nn8 <- pick_best(replicate(10, fit_one_nnet(size = 8, decay = 0.001),
                           simplify = FALSE))
pred_nn8_tr <- unscale_y(as.numeric(predict(nn8, tr$X)), tr$scaler)
pred_nn8_te <- unscale_y(as.numeric(predict(nn8, te$X)), tr$scaler)
mse_nn8 <- c(train = mse(train$price, pred_nn8_tr),
             test  = mse(test$price,  pred_nn8_te))

set.seed(5981)
nn16 <- pick_best(replicate(8, fit_one_nnet(size = 16, decay = 0.002),
                            simplify = FALSE))
pred_nn16_tr <- unscale_y(as.numeric(predict(nn16, tr$X)), tr$scaler)
pred_nn16_te <- unscale_y(as.numeric(predict(nn16, te$X)), tr$scaler)
mse_nn16 <- c(train = mse(train$price, pred_nn16_tr),
              test  = mse(test$price,  pred_nn16_te))

set.seed(5982)
nn_wide <- pick_best(replicate(6, fit_one_nnet(size = 24, decay = 1e-5),
                               simplify = FALSE))
pred_wide_tr <- unscale_y(as.numeric(predict(nn_wide, tr$X)), tr$scaler)
pred_wide_te <- unscale_y(as.numeric(predict(nn_wide, te$X)), tr$scaler)
mse_wide <- c(train = mse(train$price, pred_wide_tr),
              test  = mse(test$price,  pred_wide_te))

list(nn8 = mse_nn8, nn16 = mse_nn16, nn_wide = mse_wide)
## $nn8
##    train     test 
## 1246.668 1659.976 
## 
## $nn16
##    train     test 
## 1071.003 1933.109 
## 
## $nn_wide
##     train      test 
##  775.9467 2536.3354

 

Train and Test MSE

 

## # A tibble: 6 × 5
##   method                                   chapter  train_MSE test_MSE test_RMSE
##   <chr>                                    <chr>        <dbl>    <dbl>     <dbl>
## 1 OLS, hand-built φ (corner + interaction) previou…     1379.    1502.      38.8
## 2 NN, 8 hidden units                       this ch…     1247.    1660.      40.7
## 3 OLS, cubic in size                       previou…     1547.    1677.      40.9
## 4 NN, 16 hidden units                      this ch…     1071.    1933.      44.0
## 5 NN, 24 hidden units, tiny decay          this ch…      776.    2536.      50.4
## 6 OLS, raw features                        previou…     2213.    2622.      51.2

 

What raw OLS missed

The results we obtain are not surprising. The “hand-built” \(\phi\) is the best, since it captures exactly the corner points. Obviously, this is not easy to guess in practice (when one does not know whether even to expect a corner). An experienced statistician might pick it up by looking at the residual plots.

A small net with a little weight decay is quite close to the hand-picked OLS, without being told the columns which shows how well the NN learned the shape of \(phi\).

This method is followed by a cubic in size which fakes a bend, but misses the interaction. As expected Raw OLS is the worst. One slope cannot be a floor. Also, a wide net with almost no decay is often best on the training MSE but it performs worse on test MSE.