Random Forests in R

Load randomForest and hold out a test set.  

library(randomForest)
## randomForest 4.7-1.2
## Type rfNews() to see new features/changes/bug fixes.
## 
## Attaching package: 'randomForest'
## The following object is masked from 'package:ggplot2':
## 
##     margin
n = nrow(whitewines)
ntest = round(n * 0.3)
set.seed(598)
test.id = sample(1:n, ntest)

Model Fitting

Predict quality from every other column.

rf.wines = randomForest(quality ~ ., data = whitewines[-test.id,
    ], importance = T, ntree = 400)

mtry is the number of variables eligible at each split.

rf.wines$mtry
## [1] 3

Here mtry is 3, which matches the usual regression default \(\lfloor p/3 \rfloor\).

Test-set mean squared error:

yhat.test = predict(rf.wines, whitewines[test.id, ])

sum((whitewines$quality[test.id] - yhat.test)^2)/length(test.id)
## [1] 0.4129956

There are two training-side errors: the error of predict() on the training rows, and the out-of-bag error stored in the forest.

yhat.train = predict(rf.wines, whitewines[-test.id, ])
sum((whitewines$quality[-test.id] - yhat.train)^2)/(n - ntest)
## [1] 0.07462265

The in-sample training error is much smaller than the test error.

sum((whitewines$quality[-test.id] - rf.wines$predicted)^2)/(n -
    ntest)
## [1] 0.3717869

The OOB error is closer to the test error and is a more reliable estimate of generalization performance.   How often each observation was left out of a bootstrap sample (first 10 rows):

rf.wines$oob.times[1:10]
##  [1] 134 139 136 156 153 146 127 151 167 149

  

Performance vs. Number of Trees

par(mfrow = c(1, 2))
plot(rf.wines)

The plot is OOB MSE against the number of trees. Error usually drops quickly and then levels off. Extra trees rarely make the forest worse; they may simply be unnecessary.

Variable Importance

Because we set importance = TRUE, the forest stores an importance table.

rf.wines$importance
##                         %IncMSE IncNodePurity
## fixed.acidity        0.06065622      160.4879
## volatile.acidity     0.17216691      268.8119
## citric.acid          0.08534572      180.0529
## residual.sugar       0.14440143      190.9511
## chlorides            0.11077918      214.0642
## free.sulfur.dioxide  0.13763805      267.8467
## total.sulfur.dioxide 0.09552685      193.8788
## density              0.24357653      303.1417
## pH                   0.08025148      175.5309
## sulphates            0.04998015      152.2842
## alcohol              0.35362770      407.9546

%IncMSE is permutation importance. IncNodePurity is the accumulated RSS gain. Scaled and unscaled versions are available, with standard errors for the permutation measure.

importance(rf.wines, scale = F)
##                       %IncMSE IncNodePurity
## fixed.acidity        37.26205      160.4879
## volatile.acidity     66.72695      268.8119
## citric.acid          40.68951      180.0529
## residual.sugar       41.65007      190.9511
## chlorides            45.14400      214.0642
## free.sulfur.dioxide  58.44274      267.8467
## total.sulfur.dioxide 39.10194      193.8788
## density              35.64388      303.1417
## pH                   41.87941      175.5309
## sulphates            36.13850      152.2842
## alcohol              53.21555      407.9546
par(mfrow = c(1, 2))
varImpPlot(rf.wines, type = 1)
varImpPlot(rf.wines, type = 2)