To illustrate the ideas in this chapter, we will predict
house prices from a set of features including
size, beds, zip, and a
wealth index. This is a simulated example for illustration
purposes, and the data can be found and downloaded in Section 5.3.
The response \(y\) (house price) is obviously continuous, so we will fit linear and nonlinear regression models, this is the specified \(\phi\) from the previous chapters, and then we will employ a neural net to learn \(\phi_c\) using a squared error loss. In the end we will compare the results on the testing set.
Specifically, we will fit:
OLS on the raw features, \(\phi(x)=x\);
OLS on a hand-built \(\phi\) (a corner in size, and interaction \(\mathrm{beds}\times\mathrm{wealth}\));
OLS on a cubic in size plus the other raw
features;
a one-hidden-layer net with a linear last layer, two widths, with weight decay.
We will report \(\frac1n\sum_i(\hat y_i-y_i)^2\) on a test set.
library(tidyverse)
library(nnet)
set.seed(598)
relu <- function(z) pmax(z, 0)
mse <- function(y, yhat) mean((y - yhat)^2)
houses <- read.csv("houses.csv", header = TRUE) %>%
mutate(zip = factor(zip))
zip must be a factor which means that it will enter the
regression as a et of dummy variables.
We start by investigating the pairwise relationship of the response
vs. size and wealth grouped by zip code.
houses %>%
ggplot(aes(size, price, colour = beds)) +
geom_point(alpha = 0.55, size = 1.6) +
geom_smooth(method = "loess", se = FALSE, colour = "black", linewidth = 0.7) +
geom_vline(xintercept = 1100, linetype = 2, colour = "grey50") +
scale_colour_viridis_c() +
labs(
title = "Price vs size: a floor, then a slope",
subtitle = "Dashed line: 1100 ft², where the size effect starts in the hidden law",
x = "size (ft²)", y = "price ($1000s)",
colour = "beds"
) +
theme_minimal(base_size = 12)
We can see that the straight line does not have a very good fit. There is a corner/elbow on the lower left of the plot.
houses %>%
ggplot(aes(wealth, price)) +
geom_point(alpha = 0.4) +
geom_smooth(method = "lm", se = FALSE, colour = "steelblue") +
facet_wrap(~zip) +
labs(
title = "Wealth vs price, by zip",
x = "neighbourhood wealth (index)", y = "price ($1000s)"
) +
theme_minimal(base_size = 12)
We start by scaling the numeric columns in the training set. For
nnet, we also need to scale \(y\) since the nnet package is
unstable when the output lives on a \(10^2\)–\(10^3\) scale.
index <- sample(seq_len(nrow(houses)), size = floor(0.75 * nrow(houses)))
train <- houses[index, ]
test <- houses[-index, ]
make_xy <- function(df, scaler = NULL) {
mm <- model.matrix(price ~ size + beds + zip + wealth, data = df)[, -1]
if (is.null(scaler)) {
scaler <- list(
x_center = colMeans(mm),
x_scale = pmax(apply(mm, 2, sd), 1e-8),
y_center = mean(df$price),
y_scale = sd(df$price)
)
}
X <- scale(mm, center = scaler$x_center, scale = scaler$x_scale)
y_sc <- (df$price - scaler$y_center) / scaler$y_scale
list(X = X, y = df$price, y_sc = y_sc, scaler = scaler)
}
unscale_y <- function(y_sc, scaler) scaler$y_center + scaler$y_scale * y_sc
tr <- make_xy(train)
te <- make_xy(test, scaler = tr$scaler)
We are now ready to run the different types of “regressions”. For each model, we keep the MSE on the testing set to compare in the end.
This is the usual OLS which we means that we specify \(f_\beta(x)=\beta^\top x\) on size, beds, zip dummies, and wealth.
fit_raw <- lm(price ~ size + beds + zip + wealth, data = train)
pred_raw_tr <- predict(fit_raw, newdata = train)
pred_raw_te <- predict(fit_raw, newdata = test)
mse_raw <- c(train = mse(train$price, pred_raw_tr),
test = mse(test$price, pred_raw_te))
coef(fit_raw)
## (Intercept) size beds zip94301 zip94305 zip95112
## -261.6230314 0.2630489 51.3892150 -3.5935366 45.3503352 6.5097652
## wealth
## 199.6762570
mse_raw
## train test
## 2212.909 2622.138
In the simulated data, we have a a floor on size at \(1100\) ft\(^2\), and \(\mathrm{beds}\times\mathrm{wealth}\). So, we fit an OLS that matches this specific behavior:
train_phi <- train %>%
mutate(
size_corner = relu(size - 1100),
beds_wealth = beds * wealth
)
test_phi <- test %>%
mutate(
size_corner = relu(size - 1100),
beds_wealth = beds * wealth
)
fit_phi <- lm(
price ~ size_corner + beds + zip + wealth + beds_wealth,
data = train_phi
)
pred_phi_tr <- predict(fit_phi, newdata = train_phi)
pred_phi_te <- predict(fit_phi, newdata = test_phi)
mse_phi <- c(train = mse(train$price, pred_phi_tr),
test = mse(test$price, pred_phi_te))
coef(fit_phi)
## (Intercept) size_corner beds zip94301 zip94305 zip95112
## 66.644408 0.281340 26.203743 -3.322533 49.660671 10.872722
## wealth beds_wealth
## 118.070683 26.133155
mse_phi
## train test
## 1378.671 1502.382
size; Raw in other
Features
fit_poly <- lm(price ~ poly(size, 3) + beds + zip + wealth, data = train)
pred_poly_tr <- predict(fit_poly, newdata = train)
pred_poly_te <- predict(fit_poly, newdata = test)
mse_poly <- c(train = mse(train$price, pred_poly_tr),
test = mse(test$price, pred_poly_te))
mse_poly
## train test
## 1547.147 1676.755
## # A tibble: 6 × 5
## method chapter train_MSE test_MSE test_RMSE
## <chr> <chr> <dbl> <dbl> <dbl>
## 1 OLS, hand-built φ (corner + interaction) previou… 1379. 1502. 38.8
## 2 NN, 8 hidden units this ch… 1247. 1660. 40.7
## 3 OLS, cubic in size previou… 1547. 1677. 40.9
## 4 NN, 16 hidden units this ch… 1071. 1933. 44.0
## 5 NN, 24 hidden units, tiny decay this ch… 776. 2536. 50.4
## 6 OLS, raw features previou… 2213. 2622. 51.2
The results we obtain are not surprising. The “hand-built” \(\phi\) is the best, since it captures exactly the corner points. Obviously, this is not easy to guess in practice (when one does not know whether even to expect a corner). An experienced statistician might pick it up by looking at the residual plots.
A small net with a little weight decay is quite close to the hand-picked OLS, without being told the columns which shows how well the NN learned the shape of \(phi\).
This method is followed by a cubic in size which fakes a bend, but misses the interaction. As expected Raw OLS is the worst. One slope cannot be a floor. Also, a wide net with almost no decay is often best on the training MSE but it performs worse on test MSE.