6.4 Conclusion
A regression tree approximates a mean function by a piecewise-constant fit on hyper-rectangles. The rectangles are built by greedy binary splits that minimize squared error, and each leaf is the average of \(Y\) in that cell. One tree is easy to read and does not require a global linear form, but it is a high-variance predictor: a small change in the data can change the split sequence. The usual remedy is to grow a large tree and prune it with the cost-complexity criterion \(R_\alpha(T)=R(T)+\alpha|T|\), choosing \(\alpha\) by cross-validation (minimum CV error or the 1-SE rule). Ensembles trade the single-tree picture for accuracy. Bagging averages trees grown on bootstrap samples; a random forest does the same and, at each split, searches only \(m\) of the \(p\) predictors so that the trees are less correlated. Boosting grows trees sequentially, at each step fitting the current residuals and adding the new tree with a learning rate \(\lambda\). In short: use one pruned tree when you need a short list of rules; use a forest or a boosted ensemble when the goal is prediction.