Gradient Boosting Machines in R

A GBM is a sum of trees built sequentially. For squared error (distribution = "gaussian"), tree \(t\) is fit to the current residuals and then added with a learning rate \(\lambda\):

\[ F_t(x) = F_{t-1}(x) + \lambda\,\hat h_t(x). \]

The arguments that matter for gbm are:

  • distribution: loss. Use "gaussian" for squared error.

  • n.trees: number of trees \(T\) (default 100). Too small underfits; too large can overfit if \(\lambda\) is not small.

  • shrinkage: learning rate \(\lambda\) (default 0.1). Smaller \(\lambda\) usually needs a larger n.trees.

  • interaction.depth: size of each tree (default 1, a stump). Depth \(d\) allows interactions up to order \(d\).

  • bag.fraction: fraction of the training rows used to fit each tree (default 0.5). This is stochastic gradient boosting; 1 uses all rows every time.

  • cv.folds: folds for choosing n.trees (default 0, no CV error).

library(gbm)
## Loaded gbm 2.3.1
## This version of gbm is no longer under development. Consider transitioning to gbm3, https://github.com/gbm-developers/gbm3

The first specification uses no shrinkage and no subsampling, so it is close to unregularized stagewise fitting. That is a poor default in practice; we use it only as a baseline.

gbm.wines = gbm(quality ~ ., data = whitewines[-test.id, ], distribution = "gaussian",
    n.trees = 100, shrinkage = 1, interaction.depth = 3, bag.fraction = 1,
    cv.folds = 5)
gbm.wines
## gbm(formula = quality ~ ., distribution = "gaussian", data = whitewines[-test.id, 
##     ], n.trees = 100, interaction.depth = 3, shrinkage = 1, bag.fraction = 1, 
##     cv.folds = 5)
## A gradient boosted model with gaussian loss function.
## 100 iterations were performed.
## The best cross-validation iteration was 8.
## There were 11 predictors of which 10 had non-zero influence.

  Plot the CV error against the number of trees:

opt.size = gbm.perf(gbm.wines)

gbm.perf picks a number of trees from the CV curve. With \(\lambda=1\), that optimum is often early: later trees add little or start to overfit.

Test MSE as a function of the number of trees:

size = 1:gbm.wines$n.trees
test.err = rep(0, length(size))
for (i in 1:length(size)) {
    y.pred = predict(gbm.wines, whitewines[test.id, ], n.trees = size[i])
    test.err[i] = sum((whitewines$quality[test.id] - y.pred)^2)
}
plot(test.err, type = "n")
lines(size, test.err, lwd = 2)
abline(v = opt.size, lwd = 2, lty = 2, col = "blue")

A more standard specification: small learning rate, more trees, and bag.fraction = 0.5.

gbm.wines.2 = gbm(quality ~ ., data = whitewines[-test.id, ],
    distribution = "gaussian", n.trees = 1000, shrinkage = 0.1,
    interaction.depth = 3, bag.fraction = 0.5, cv.folds = 5)

opt.size.2 = gbm.perf(gbm.wines.2)

size.2 = 1:gbm.wines.2$n.trees
test.err.2 = rep(0, length(size.2))
for (i in 1:length(size.2)) {
    y.pred.2 = predict(gbm.wines.2, whitewines[test.id, ], n.trees = size.2[i])
    test.err.2[i] = sum((whitewines$quality[test.id] - y.pred.2)^2)
}
plot(test.err.2, type = "n")
lines(size.2, test.err.2, lwd = 2)
abline(v = opt.size.2, lwd = 2, lty = 2, col = "blue")

Variable importance from the baseline fit (relative influence and a permutation measure):

par(mfrow = c(1, 2))
summary(gbm.wines, cBars = 10, method = relative.influence, las = 2)
##                                       var   rel.inf
## alcohol                           alcohol 35.531879
## volatile.acidity         volatile.acidity 13.540395
## free.sulfur.dioxide   free.sulfur.dioxide 12.034661
## residual.sugar             residual.sugar  7.791508
## density                           density  6.230189
## total.sulfur.dioxide total.sulfur.dioxide  5.006459
## pH                                     pH  4.494230
## chlorides                       chlorides  4.058416
## fixed.acidity               fixed.acidity  3.918535
## sulphates                       sulphates  3.871500
## citric.acid                   citric.acid  3.522228
summary(gbm.wines, cBars = 10, method = permutation.test.gbm,
    las = 2)

##                     var   rel.inf
## 1        residual.sugar 20.976784
## 2               density 17.275031
## 3               alcohol 15.905093
## 4   free.sulfur.dioxide  8.993392
## 5      volatile.acidity  8.822918
## 6  total.sulfur.dioxide  6.674502
## 7                    pH  6.117880
## 8             chlorides  4.568066
## 9             sulphates  4.529224
## 10        fixed.acidity  3.218505
## 11          citric.acid  2.918603