6.3b Tree-Based Regression Models in Python¶
This notebook is the Python counterpart of the R section Tree-Based Regression Models in R. The analysis is the same -- white-wine quality, single trees, cost-complexity pruning, random forests, and gradient boosting -- implemented with scikit-learn, pandas, and matplotlib.
sklearn does not reproduce tree, rpart, randomForest, or gbm line for line. Differences that change the output (a number or a plot) are discussed.
Two R packages fit trees in the notes: tree and rpart. In Python, we have one class, DecisionTreeRegressor that covers both illustration and pruning. Ensembles are RandomForestRegressor and GradientBoostingRegressor.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeRegressor, plot_tree
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.inspection import permutation_importance
from sklearn.metrics import mean_squared_error
plt.style.use("seaborn-v0_8-darkgrid")
RANDOM_STATE = 598 # same seed as the R notes
The whitewines.csv data set contains information on white variants of the Portuguese "Vinho Verde" wine.
The columns are:
(a) fixed acidity,
(b) volatile acidity,
(c) citric acid,
(d) residual sugar,
(e) chlorides,
(f) free sulfur dioxide,
(g) total sulfur dioxide,
(h) density,
(i) pH,
(j) sulphates,
(k) alcohol,
(l) quality (score between 0 and 10).
The file used here is the same semicolon-separated table as in the R notes and can be downloaded from there. The UCI page that contains a more detailed description is Wine Quality.
# sklearn wants valid Python identifiers, so we replace spaces with underscores.
whitewines = pd.read_csv("whitewines.csv", sep=";")
whitewines.columns = whitewines.columns.str.replace(" ", "_")
print("First 5 rows:")
display(whitewines.head())
print(f"\nDimensions (rows, columns): {whitewines.shape}")
First 5 rows:
| fixed_acidity | volatile_acidity | citric_acid | residual_sugar | chlorides | free_sulfur_dioxide | total_sulfur_dioxide | density | pH | sulphates | alcohol | quality | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 7.0 | 0.27 | 0.36 | 20.7 | 0.045 | 45.0 | 170.0 | 1.0010 | 3.00 | 0.45 | 8.8 | 6 |
| 1 | 6.3 | 0.30 | 0.34 | 1.6 | 0.049 | 14.0 | 132.0 | 0.9940 | 3.30 | 0.49 | 9.5 | 6 |
| 2 | 8.1 | 0.28 | 0.40 | 6.9 | 0.050 | 30.0 | 97.0 | 0.9951 | 3.26 | 0.44 | 10.1 | 6 |
| 3 | 7.2 | 0.23 | 0.32 | 8.5 | 0.058 | 47.0 | 186.0 | 0.9956 | 3.19 | 0.40 | 9.9 | 6 |
| 4 | 7.2 | 0.23 | 0.32 | 8.5 | 0.058 | 47.0 | 186.0 | 0.9956 | 3.19 | 0.40 | 9.9 | 6 |
Dimensions (rows, columns): (4898, 12)
A Simple Regression Tree¶
In R the two-predictor illustration uses the tree package (tree(), plot(), partition.tree()). sklearn's DecisionTreeRegressor is the analogue of both tree and rpart; there is no separate partition.tree.
The syntax is not a formula. We pass a design matrix X and a response vector y.
# R: tree(quality ~ alcohol + volatile.acidity, data=whitewines)
feature_names = ["alcohol", "volatile_acidity"]
X_simple = whitewines[feature_names]
y = whitewines["quality"]
# sklearn grows until leaves are pure unless we stop it.
# tree() in R uses different defaults, so these two knobs are only
# to keep the illustration small and readable, not to match rpart's CP path.
simple_tree = DecisionTreeRegressor(
max_depth=3,
min_impurity_decrease=0.008,
random_state=RANDOM_STATE,
)
simple_tree.fit(X_simple, y)
plt.figure(figsize=(10, 6))
plot_tree(
simple_tree,
feature_names=list(X_simple.columns),
filled=True,
rounded=True,
fontsize=10,
)
plt.title("Simple regression tree for wine quality")
plt.show()
Rectangular partition¶
R draws the partition with partition.tree(). sklearn has no helper, so the next cell is a small translation of the tree package geometry: walk the fitted tree and draw the cut segments in the (alcohol, volatile_acidity) plane.
Points are colored by quality, as in the R figure (not by the vertical coordinate).
from matplotlib.collections import LineCollection
def _partition_segments(regressor, bbox, node=0):
"""Walk a two-predictor sklearn tree using child indices, not R's DFS vectors.
bbox = [x_min, y_min, x_max, y_max]. Left child is {feature <= threshold}.
"""
tree = regressor.tree_
if tree.children_left[node] == -1: # leaf
return []
feat = tree.feature[node]
cut = tree.threshold[node]
left = tree.children_left[node]
right = tree.children_right[node]
x_min, y_min, x_max, y_max = bbox
segs = []
if feat == 0: # alcohol (x-axis)
segs.append([(cut, y_min), (cut, y_max)])
segs += _partition_segments(regressor, [x_min, y_min, cut, y_max], left)
segs += _partition_segments(regressor, [cut, y_min, x_max, y_max], right)
elif feat == 1: # volatile_acidity (y-axis)
segs.append([(x_min, cut), (x_max, cut)])
segs += _partition_segments(regressor, [x_min, y_min, x_max, cut], left)
segs += _partition_segments(regressor, [x_min, cut, x_max, y_max], right)
else:
raise ValueError("split on a variable other than the two plotted predictors")
return segs
def partition_tree(regressor, data, y, ax=None, **scatter_kw):
"""Scatter two predictors and overlay the tree partition (R: partition.tree)."""
x_min, y_min = data.min().to_numpy()
x_max, y_max = data.max().to_numpy()
pad_x = 0.025 * (x_max - x_min)
pad_y = 0.025 * (y_max - y_min)
bbox = [x_min - pad_x, y_min - pad_y, x_max + pad_x, y_max + pad_y]
segs = _partition_segments(regressor, bbox)
if ax is None:
_, ax = plt.subplots()
ax.scatter(data.iloc[:, 0], data.iloc[:, 1], c=y, **scatter_kw)
ax.set_xlabel(data.columns[0])
ax.set_ylabel(data.columns[1])
ax.add_collection(LineCollection(segs, colors="k"))
ax.autoscale()
return ax
fig, ax = plt.subplots(figsize=(10, 6))
partition_tree(simple_tree, X_simple, y, ax, cmap="Greys", alpha=0.6, s=18)
ax.set_title("Partition in the (alcohol, volatile_acidity) plane")
plt.show()
R also calls prune.tree(..., best=4) to force four leaves. The sklearn analogue is to pick a ccp_alpha (next section) or to set max_leaf_nodes=4. We skip that extra plot; the pruned full-predictor tree below is the one that corresponds to prune() in rpart.
Fitting and Pruning a Regression Tree {-}¶
Unlike rpart, sklearn does not run cross-validation or prune while fitting. DecisionTreeRegressor() with default arguments grows a large tree $T_{\max}$ (the cp = 0 fit wine.tree2 in the notes). Pruning is a second step, using ccp_alpha.
sklearn's ccp_alpha is $\alpha$ itself in $R_\alpha(T)=R(T)+\alpha|T|$. The path of trees is the same idea but the printed numbers are not on the same scale.
X = whitewines.drop("quality", axis=1)
y = whitewines["quality"]
# Default sklearn tree = large T_max (R: control=list(cp=0)).
wine_tree_max = DecisionTreeRegressor(random_state=RANDOM_STATE)
wine_tree_max.fit(X, y)
# Full plot of T_max is unreadable; show the top layers only.
plt.figure(figsize=(15, 8))
plot_tree(
wine_tree_max,
max_depth=3,
feature_names=list(X.columns),
filled=True,
rounded=True,
fontsize=8,
)
plt.title("Large tree $T_{max}$ (top 3 levels)")
plt.show()
Reading the Tree¶
Start at the root and read down. The root is the mean of quality over the whole sample. The first split is on alcohol. The left child is the lower-alcohol rows (lower mean); the right child is the rest. Leaves sit at the bottom; shading tracks the predicted mean. Exact means will match this fitted sklearn tree. The greedy search is the same idea, the tie-breaking and default stopping rules are not.
Cross-validation and the $\alpha$ path¶
rpart builds the cost-complexity path and the CV table inside rpart(). In sklearn we ask for the path first, then run $K$-fold CV ourselves.
cost_complexity_pruning_path returns the increasing sequence of effective $\alpha$ values (the $\alpha_i$ from weakest-link pruning) and the corresponding total leaf impurity.
path = wine_tree_max.cost_complexity_pruning_path(X, y)
# Full path can have hundreds of alphas (here often ~900).
# 10-fold CV on every value is too slow for a lecture notebook.
# Keep ~40 alphas spaced along the path, including the endpoints.
ccp_alphas_full, impurities_full = path.ccp_alphas, path.impurities
idx = np.unique(np.linspace(0, len(ccp_alphas_full) - 1, 40).astype(int))
ccp_alphas = ccp_alphas_full[idx]
impurities = impurities_full[idx]
plt.figure(figsize=(8, 5))
plt.plot(ccp_alphas, impurities, marker="o", drawstyle="steps-post")
plt.xlabel(r"effective $\alpha$")
plt.ylabel("Total impurity of leaves")
plt.title(r"Training impurity vs. effective $\alpha$")
plt.show()
Choosing $\alpha$: minimum CV error and the 1-SE rule¶
As in the notes, use the minimum cross-validation error or the one-standard-error (1-SE) rule. We run 10-fold CV over the $\alpha$ path (xval = 10 in rpart.control).
GridSearchCV reports mean negative MSE, so larger scores are better. The 1-SE tree is the simplest tree (largest $\alpha$) whose CV score is within one standard error of the best score.
cv_fold = 10
tree_cv = GridSearchCV(
DecisionTreeRegressor(random_state=RANDOM_STATE),
param_grid={"ccp_alpha": ccp_alphas},
cv=cv_fold,
scoring="neg_mean_squared_error",
n_jobs=-1,
)
tree_cv.fit(X, y)
GridSearchCV(cv=10, estimator=DecisionTreeRegressor(random_state=598),
n_jobs=-1,
param_grid={'ccp_alpha': array([0.00000000e+00, 9.42299840e-05, 9.81562333e-05, 1.02082483e-04,
1.36109977e-04, 1.36109977e-04, 1.36109977e-04, 1.53123724e-04,
1.53123724e-04, 1.63331972e-04, 1.67519972e-04, 1.72525365e-04,
1.78644345e-04, 1.81479969e-04, 1.85604514e-04, 1.93917303e-04,
2.04164965e-04, 2.15654769e-04, 2.33331389e-04, 2.44997958e-04,
2.52952117e-04, 2.72219954e-04, 2.83562452e-04, 2.99041626e-04,
3.11108519e-04, 3.35039943e-04, 3.53362440e-04, 3.75654051e-04,
3.97070833e-04, 4.29721220e-04, 4.77469705e-04, 5.35933034e-04,
6.06219171e-04, 6.76528453e-04, 7.65618620e-04, 8.86789095e-04,
1.06515657e-03, 1.36815182e-03, 1.99701311e-03, 1.26260584e-01])},
scoring='neg_mean_squared_error')In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
Fitted attributes
DecisionTreeRegressor(ccp_alpha=np.float64(0.001997013105639954),
random_state=598)Parameters
Fitted attributes
# Relative CV error, analogous to rpart's xerror
# (divide by the CV error of the largest-alpha tree = the root).
cv_mse = -tree_cv.cv_results_["mean_test_score"]
cv_mse_rel = cv_mse / cv_mse[-1]
plt.figure(figsize=(8, 5))
plt.plot(ccp_alphas, cv_mse_rel, marker="o", drawstyle="steps-post")
plt.xlabel(r"$\alpha$")
plt.ylabel("Relative CV MSE")
plt.title("Cost-complexity path (Python stand-in for plotcp)")
plt.show()
best_alpha = tree_cv.best_params_["ccp_alpha"]
best_idx = tree_cv.best_index_
best_score = tree_cv.best_score_ # negative MSE
print(f"alpha at minimum CV error: {best_alpha:.6f}")
# SE of the mean CV score. GridSearchCV's std_test_score is the
# standard deviation across folds; divide by sqrt(K) for the SE.
cv_scores = tree_cv.cv_results_["mean_test_score"]
best_se = tree_cv.cv_results_["std_test_score"][best_idx] / np.sqrt(cv_fold)
threshold = best_score - best_se
# ccp_alphas increase with simplicity. Take the last alpha still within 1 SE.
one_se_idx = np.where(cv_scores >= threshold)[0][-1]
one_se_alpha = ccp_alphas[one_se_idx]
print(f"1-SE alpha: {one_se_alpha:.6f}")
alpha at minimum CV error: 0.001997 1-SE alpha: 0.001997
wine_tree_pruned = DecisionTreeRegressor(
random_state=RANDOM_STATE, ccp_alpha=one_se_alpha
)
wine_tree_pruned.fit(X, y)
plt.figure(figsize=(10, 6))
plot_tree(
wine_tree_pruned,
feature_names=list(X.columns),
filled=True,
rounded=True,
fontsize=8,
)
plt.title(rf"Pruned tree (1-SE $\alpha$ = {one_se_alpha:.4f})")
plt.show()
After the tree is fit, predict() produces fitted values on new or existing rows.
Random Forests in Python¶
Load the forest estimator and hold out a test set. R used sample() with seed 598 and a 30% holdout. train_test_split(..., random_state=598, test_size=0.3) is the same fraction and the same seed integer, but it is not the same partition as R's sample(), so the MSEs will not match the knitted R output exactly.
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=RANDOM_STATE
)
print(f"Train rows: {X_train.shape[0]}, test rows: {X_test.shape[0]}")
Train rows: 3428, test rows: 1470
Model fitting¶
RandomForestRegressor is the analogue of randomForest().
Note that oob_score_ itself is an OOB $R^2$, not an MSE. For a number comparable to the R notes we compute MSE from oob_prediction_.
rf_wines = RandomForestRegressor(
n_estimators=400,
max_features=1 / 3, # p/3, the usual regression default
oob_score=True,
random_state=RANDOM_STATE,
n_jobs=-1,
)
rf_wines.fit(X_train, y_train)
print(f"max_features used: {rf_wines.max_features}")
print(f"mtry in the R notes was 3; here p/3 = {X_train.shape[1] / 3:.2f}")
max_features used: 0.3333333333333333 mtry in the R notes was 3; here p/3 = 3.67
Test-set mean squared error:
test_mse = mean_squared_error(y_test, rf_wines.predict(X_test))
print(f"Test MSE: {test_mse:.4f}")
Test MSE: 0.3912
There are two training-side errors: the error of predict() on the training rows, and the out-of-bag error stored on the forest.
train_mse = mean_squared_error(y_train, rf_wines.predict(X_train))
oob_mse = mean_squared_error(y_train, rf_wines.oob_prediction_)
print(f"Training MSE (in-sample predict): {train_mse:.4f}")
print(f"OOB MSE: {oob_mse:.4f}")
Training MSE (in-sample predict): 0.0498 OOB MSE: 0.3700
The in-sample training error is much smaller than the test error. The OOB error is closer to the test error and is a more reliable estimate of generalization performance.
Performance vs. Number of Trees¶
R's plot(rf) is OOB MSE against the number of trees. sklearn does not produce that plot. The next cell rebuilds a running OOB average from the individual trees. It uses _generate_unsampled_indices, an internal helper that mirrors how the forest drew each bootstrap sample. Treat it as a plotting device, not as public API.
from sklearn.ensemble._forest import _generate_unsampled_indices
n_samples = X_train.shape[0]
n_trees = rf_wines.n_estimators
oob_preds = np.full((n_samples, n_trees), np.nan)
for i, tree in enumerate(rf_wines.estimators_):
oob_idx = _generate_unsampled_indices(tree.random_state, n_samples, n_samples, None)
oob_preds[oob_idx, i] = tree.predict(X_train.iloc[oob_idx].to_numpy())
oob_count = np.cumsum(~np.isnan(oob_preds), axis=1)
with np.errstate(divide="ignore", invalid="ignore"):
oob_mean = np.nancumsum(oob_preds, axis=1) / oob_count
oob_mse_path = np.nanmean((oob_mean - y_train.to_numpy()[:, None]) ** 2, axis=0)
plt.figure(figsize=(8, 5))
plt.plot(np.arange(1, n_trees + 1), oob_mse_path)
plt.xlabel("Number of trees")
plt.ylabel("OOB MSE")
plt.title("Random forest OOB error vs. number of trees")
plt.show()
The error usually drops quickly and then levels off. Extra trees rarely make the forest worse; they may simply be unnecessary.
Variable Importance¶
R reports both %IncMSE (permutation) and IncNodePurity (RSS / impurity gain). rf_wines.feature_importances_ is the impurity measure only. Permutation importance is computed separately with permutation_importance on the OOB-style held-out rows -- here we use the test set for a clean comparison.
impurity = pd.Series(rf_wines.feature_importances_, index=X.columns).sort_values()
perm = permutation_importance(
rf_wines, X_test, y_test,
n_repeats=10,
random_state=RANDOM_STATE,
n_jobs=-1,
)
perm_imp = pd.Series(perm.importances_mean, index=X.columns).sort_values()
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].barh(perm_imp.index, perm_imp.values)
axes[0].set_title("Permutation importance")
axes[0].set_xlabel("Increase in test MSE")
axes[1].barh(impurity.index, impurity.values)
axes[1].set_title("Impurity importance")
axes[1].set_xlabel("Mean decrease in impurity")
fig.tight_layout()
plt.show()
Gradient Boosting Machines in Python¶
A GBM is a sum of trees built sequentially. For squared error, tree $t$ is fit to the current residuals and added with a learning rate $\lambda$:
$$ F_t(x) = F_{t-1}(x) + \lambda\,\hat h_t(x). $$
The gbm arguments and their sklearn names:
sklearn (GradientBoostingRegressor) |
Role |
|---|---|
loss="squared_error" (default) |
squared error |
n_estimators |
number of trees $T$ |
learning_rate |
$\lambda$ (default 0.1 in both) |
max_depth |
R default is 1 (stumps); sklearn default is 3 |
subsample |
R default 0.5; sklearn default 1.0 (no subsampling) |
| none | gbm.perf uses CV; sklearn has no built-in CV curve. We plot test MSE vs. $T$ with staged_predict. |
The first specification below uses no shrinkage and no subsampling, so it is close to unregularized stagewise fitting. That is a poor default; we use it only as a baseline, matching the R notes.
gbm_wines = GradientBoostingRegressor(
n_estimators=100,
learning_rate=1.0,
max_depth=3,
subsample=1.0,
random_state=RANDOM_STATE,
)
gbm_wines.fit(X_train, y_train)
GradientBoostingRegressor(learning_rate=1.0, random_state=598)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
Fitted attributes
Number of trees¶
gbm.perf picks $T$ from a CV curve. sklearn does not. train_score_ is the training loss after each tree; staged_predict gives the test path. With $\lambda=1$, the test-MSE minimum is often early.
train_err = gbm_wines.train_score_ # training MSE path
test_err = np.array([
mean_squared_error(y_test, y_hat)
for y_hat in gbm_wines.staged_predict(X_test)
])
opt_size = int(np.argmin(test_err) + 1)
plt.figure(figsize=(10, 6))
plt.plot(np.arange(1, len(train_err) + 1), train_err, label="Training MSE")
plt.plot(np.arange(1, len(test_err) + 1), test_err, label="Test MSE")
plt.axvline(opt_size, linestyle="--", label=f"Min test MSE at T = {opt_size}")
plt.xlabel("Number of trees")
plt.ylabel("MSE")
plt.title(r"GBM baseline ($\lambda = 1$, no subsampling)")
plt.legend()
plt.show()
A more standard specification: small learning rate, more trees, and subsample=0.5 (bag.fraction = 0.5).
gbm_wines2 = GradientBoostingRegressor(
n_estimators=1000,
learning_rate=0.1,
max_depth=3,
subsample=0.5,
random_state=RANDOM_STATE,
)
gbm_wines2.fit(X_train, y_train)
train_err2 = gbm_wines2.train_score_
test_err2 = np.array([
mean_squared_error(y_test, y_hat)
for y_hat in gbm_wines2.staged_predict(X_test)
])
opt_size2 = int(np.argmin(test_err2) + 1)
plt.figure(figsize=(10, 6))
plt.plot(np.arange(1, len(train_err2) + 1), train_err2, label="Training MSE")
plt.plot(np.arange(1, len(test_err2) + 1), test_err2, label="Test MSE")
plt.axvline(opt_size2, linestyle="--", label=f"Min test MSE at T = {opt_size2}")
plt.xlabel("Number of trees")
plt.ylabel("MSE")
plt.title(r"GBM with $\lambda = 0.1$ and subsample $= 0.5$")
plt.legend()
plt.show()
Variable Importance¶
feature_importances_ on a GBM is the analogue of gbm's relative influence (normalized total improvement from splits on that variable). A permutation measure is available as in the forest section; we plot both on the second, more standard fit.
rel_inf = pd.Series(gbm_wines2.feature_importances_, index=X.columns).sort_values()
perm_gbm = permutation_importance(
gbm_wines2, X_test, y_test,
n_repeats=10,
random_state=RANDOM_STATE,
n_jobs=-1,
)
perm_gbm_s = pd.Series(perm_gbm.importances_mean, index=X.columns).sort_values()
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].barh(rel_inf.index, rel_inf.values)
axes[0].set_title("Relative influence")
axes[0].set_xlabel("Importance")
axes[1].barh(perm_gbm_s.index, perm_gbm_s.values)
axes[1].set_title("Permutation importance")
axes[1].set_xlabel("Increase in test MSE")
fig.tight_layout()
plt.show()