Classification Trees & Ensemble Methods (Python Code)¶
In this notebook we present the Python code for all the R chunks in this Chapter.
Setup¶
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pathlib import Path
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text
from sklearn.ensemble import (
BaggingClassifier,
RandomForestClassifier,
AdaBoostClassifier,
GradientBoostingClassifier,
GradientBoostingRegressor,
)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# xgboost needs an OpenMP runtime (libomp / libgomp). If you do not have thin then skip it.
try:
import xgboost as xgb
HAS_XGBOOST = True
except Exception:
xgb = None
HAS_XGBOOST = False
plt.rcParams.update({
"figure.figsize": (6, 4.5),
"axes.grid": False,
})
def decision_boundary(ax, predict_fn, xlim, ylim, n=201, colors=("darkorange", "darkblue")):
"""Shade predicted class on a 2-d grid and draw the 0.5 contour."""
xx, yy = np.meshgrid(
np.linspace(*xlim, n),
np.linspace(*ylim, n),
)
grid = np.column_stack([xx.ravel(), yy.ravel()])
zz = np.asarray(predict_fn(grid)).reshape(xx.shape)
ax.contourf(xx, yy, zz, levels=[-0.5, 0.5, 1.5],
colors=colors, alpha=0.18)
ax.contour(xx, yy, zz, levels=[0.5], colors="black", linewidths=1)
return xx, yy, zz
Titanic Tree¶
We use the same Titanic data as in the R example:
DATA = Path("train.csv")
cols = ["Survived", "Pclass", "Sex", "Age", "SibSp", "Parch"]
titanic = pd.read_csv(DATA, usecols=cols)
titanic["Sex"] = (titanic["Sex"] == "male").astype(int) # rpart dummy-codes this internally
titanic_cc = titanic.dropna(subset=cols).copy()
X_all = titanic_cc[["Pclass", "Sex", "Age", "SibSp", "Parch"]]
y_all = titanic_cc["Survived"].astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(
X_all, y_all, test_size=0.30, random_state=10, stratify=y_all
)
# Full-data tree (the rpart.plot figure)
tree_full = DecisionTreeClassifier(random_state=1, ccp_alpha=0.02)
tree_full.fit(X_all, y_all)
fig, ax = plt.subplots(figsize=(12, 7))
plot_tree(
tree_full,
feature_names=list(X_all.columns),
class_names=["died", "survived"],
filled=True,
impurity=True,
proportion=True,
ax=ax,
)
ax.set_title("Pruned classification tree (ccp_alpha = 0.02)")
plt.tight_layout()
plt.show()
print(export_text(tree_full, feature_names=list(X_all.columns)))
|--- Sex <= 0.50 | |--- Pclass <= 2.50 | | |--- class: 1 | |--- Pclass > 2.50 | | |--- class: 0 |--- Sex > 0.50 | |--- class: 0
maj = int(y_tr.value_counts().idxmax())
base_err = np.mean(y_te.to_numpy() != maj)
tree_hold = DecisionTreeClassifier(random_state=1, ccp_alpha=0.02)
tree_hold.fit(X_tr, y_tr)
tree_err_tr = 1 - accuracy_score(y_tr, tree_hold.predict(X_tr))
tree_err_te = 1 - accuracy_score(y_te, tree_hold.predict(X_te))
logit = LogisticRegression(max_iter=1000)
logit.fit(X_tr, y_tr)
glm_err_te = 1 - accuracy_score(y_te, logit.predict(X_te))
pd.DataFrame({
"method": [
"Majority class (holdout)",
"Pruned tree (train)",
"Pruned tree (holdout)",
"Logistic regression (holdout)",
],
"misclassification": np.round(
[base_err, tree_err_tr, tree_err_te, glm_err_te], 3
),
})
| method | misclassification | |
|---|---|---|
| 0 | Majority class (holdout) | 0.405 |
| 1 | Pruned tree (train) | 0.194 |
| 2 | Pruned tree (holdout) | 0.219 |
| 3 | Logistic regression (holdout) | 0.214 |
Circle Simulation and a Single Tree¶
rng = np.random.default_rng(1)
n = 500
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
p = np.where(x1**2 + x2**2 < 0.6, 0.9, 0.1)
y = rng.binomial(1, p)
fig, ax = plt.subplots()
ax.scatter(x1, x2, c=np.where(y == 1, "darkblue", "darkorange"), s=18, edgecolors="none")
circ = plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2)
ax.add_patch(circ)
ax.set_aspect("equal", "box")
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.set_title("Circle simulation")
plt.show()
X_circ = np.column_stack([x1, x2])
tree_circ = DecisionTreeClassifier(random_state=1)
tree_circ.fit(X_circ, y)
fig, ax = plt.subplots(figsize=(10, 6))
plot_tree(tree_circ, feature_names=["x1", "x2"], class_names=["0", "1"],
filled=True, max_depth=4, ax=ax)
ax.set_title("Unpruned tree (depth capped in the plot only)")
plt.tight_layout()
plt.show()
Cost-complexity path (R: rpart.fit$cptable). Then prune at a chosen ccp_alpha.
Remark: Do not copy a cp value from the R notes into ccp_alpha.
ccp_alpha is the same idea as rpart's cp (cost-complexity pruning)
path = tree_circ.cost_complexity_pruning_path(X_circ, y)
cptable = pd.DataFrame({
"ccp_alpha": path.ccp_alphas,
"impurity_leaves": path.impurities,
})
cptable
| ccp_alpha | impurity_leaves | |
|---|---|---|
| 0 | 0.000000 | 0.000000 |
| 1 | 0.001500 | 0.003000 |
| 2 | 0.001714 | 0.006429 |
| 3 | 0.001771 | 0.020600 |
| 4 | 0.001786 | 0.024171 |
| 5 | 0.001810 | 0.031410 |
| 6 | 0.001818 | 0.038682 |
| 7 | 0.001846 | 0.042375 |
| 8 | 0.001857 | 0.046089 |
| 9 | 0.001857 | 0.053517 |
| 10 | 0.001867 | 0.060985 |
| 11 | 0.001882 | 0.064750 |
| 12 | 0.001897 | 0.072337 |
| 13 | 0.001900 | 0.076137 |
| 14 | 0.001902 | 0.083747 |
| 15 | 0.002000 | 0.089747 |
| 16 | 0.002012 | 0.093771 |
| 17 | 0.002037 | 0.097846 |
| 18 | 0.002390 | 0.112187 |
| 19 | 0.002400 | 0.116987 |
| 20 | 0.002400 | 0.121787 |
| 21 | 0.002400 | 0.124187 |
| 22 | 0.002497 | 0.129180 |
| 23 | 0.002667 | 0.134514 |
| 24 | 0.002829 | 0.143000 |
| 25 | 0.003126 | 0.146125 |
| 26 | 0.003200 | 0.149325 |
| 27 | 0.003333 | 0.152659 |
| 28 | 0.003500 | 0.156159 |
| 29 | 0.003579 | 0.163317 |
| 30 | 0.003955 | 0.171226 |
| 31 | 0.004345 | 0.179915 |
| 32 | 0.005352 | 0.201322 |
| 33 | 0.006199 | 0.219918 |
| 34 | 0.006341 | 0.226258 |
| 35 | 0.006824 | 0.239907 |
| 36 | 0.007088 | 0.246995 |
| 37 | 0.010683 | 0.268361 |
| 38 | 0.016029 | 0.284389 |
| 39 | 0.053703 | 0.499200 |
# Pick an alpha near the R notes' idea of a small prune (not the same number as cp = 0.041)
alpha = path.ccp_alphas[max(0, len(path.ccp_alphas) // 3)]
pruned = DecisionTreeClassifier(random_state=1, ccp_alpha=alpha)
pruned.fit(X_circ, y)
fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
plot_tree(pruned, feature_names=["x1", "x2"], class_names=["0", "1"],
filled=True, ax=axes[0])
axes[0].set_title(f"Pruned tree (ccp_alpha = {alpha:.4f})")
decision_boundary(axes[1], pruned.predict, (-1, 1), (-1, 1))
axes[1].scatter(x1, x2, c=np.where(y == 1, "darkblue", "darkorange"),
s=16, edgecolors="none")
axes[1].add_patch(plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2))
axes[1].set_aspect("equal", "box")
axes[1].set_title("Axis-aligned staircase vs Bayes circle")
plt.tight_layout()
plt.show()
Impurity Measures¶
R defined gini, shannon, error, and the split gain score.
def _freqs(y):
y = np.asarray(y)
_, counts = np.unique(y, return_counts=True)
return counts / counts.sum()
def gini(y):
p = _freqs(y)
return float(np.sum(p * (1 - p)))
def shannon(y):
p = _freqs(y)
p = p[p > 0]
return float(-np.sum(p * np.log(p)))
def misclass(y):
p = _freqs(y)
return float(1 - p.max())
def split_gain(left, right, measure):
left, right = np.asarray(left), np.asarray(right)
parent = np.concatenate([left, right])
n_l, n_r = len(left), len(right)
n = n_l + n_r
return measure(parent) - (n_l / n) * measure(left) - (n_r / n) * measure(right)
TL = np.repeat(1, 3)
TR = np.array([1, 1, 1, 1, 0, 0, 0])
print("Gini gain ", split_gain(TL, TR, gini))
print("Entropy gain ", split_gain(TL, TR, shannon))
print("Error gain ", split_gain(TL, TR, misclass))
Gini gain 0.07714285714285724 Entropy gain 0.13282862876456336 Error gain 5.551115123125783e-17
p = np.linspace(0, 1, 201)
g = 2 * p * (1 - p)
s = np.zeros_like(p)
mid = (p > 0) & (p < 1)
s[mid] = -p[mid] * np.log(p[mid]) - (1 - p[mid]) * np.log(1 - p[mid])
e = np.minimum(p, 1 - p)
plt.plot(p, s, lw=2, label="Entropy")
plt.plot(p, g, lw=2, label="Gini")
plt.plot(p, e, lw=2, label="Error")
plt.xlabel("p")
plt.ylabel("Impurity")
plt.legend()
plt.title("Binary impurity curves")
plt.show()
Bagging vs One Tree (diagonal strip)¶
R: rpart vs ipred::bagging(..., nbagg = 200).
rng = np.random.default_rng(2)
n = 1000
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
inside = (x1 + x2 > -0.5) & (x1 + x2 < 0.5)
y = rng.binomial(1, np.where(inside, 0.8, 0.2))
X = np.column_stack([x1, x2])
cart = DecisionTreeClassifier(random_state=2)
cart.fit(X, y)
bag = BaggingClassifier(
estimator=DecisionTreeClassifier(),
n_estimators=200,
random_state=2,
)
bag.fit(X, y)
fig, axes = plt.subplots(1, 2, figsize=(10, 4.5))
for ax, model, title in [(axes[0], cart, "CART"), (axes[1], bag, "Bagging")]:
decision_boundary(ax, model.predict, (-1, 1), (-1, 1))
ax.scatter(x1, x2, c=np.where(y == 1, "darkblue", "orange"),
s=12, edgecolors="none")
ax.set_title(title)
ax.set_xticks([])
ax.set_yticks([])
plt.tight_layout()
plt.show()
Random forest, OOB, Bayes agreement, importance¶
R: randomForest(X, y, ntree=1000, mtry=7, nodesize=10, sampsize=800).
sklearn names:
mtry→max_featuresnodesize→min_samples_leafsampsize→max_samples- OOB error →
1 - oob_score_
rng = np.random.default_rng(542)
n, p = 1000, 10
X = rng.uniform(-1, 1, size=(n, p))
inside = (X[:, 0] + X[:, 1] > -0.5) & (X[:, 0] + X[:, 1] < 0.5)
y = rng.binomial(1, np.where(inside, 0.8, 0.2))
rf = RandomForestClassifier(
n_estimators=1000,
max_features=7, # mtry
min_samples_leaf=10, # nodesize
max_samples=800, # sampsize
oob_score=True,
random_state=542,
n_jobs=-1,
)
rf.fit(X, y)
print("OOB error:", round(1 - rf.oob_score_, 4))
Xtest = rng.uniform(-1, 1, size=(n, p))
bayes = ((Xtest[:, 0] + Xtest[:, 1] > -0.5) &
(Xtest[:, 0] + Xtest[:, 1] < 0.5)).astype(int)
agree = np.mean(rf.predict(Xtest) == bayes)
print("Agreement with Bayes rule:", round(agree, 4))
imp = pd.DataFrame({
"feature": [f"x{j+1}" for j in range(p)],
"gini_importance": rf.feature_importances_,
}).sort_values("gini_importance", ascending=False)
imp
OOB error: 0.301 Agreement with Bayes rule: 0.847
| feature | gini_importance | |
|---|---|---|
| 1 | x2 | 0.204473 |
| 0 | x1 | 0.192906 |
| 6 | x7 | 0.082063 |
| 2 | x3 | 0.081233 |
| 8 | x9 | 0.080029 |
| 5 | x6 | 0.075407 |
| 4 | x5 | 0.074082 |
| 9 | x10 | 0.072909 |
| 3 | x4 | 0.070642 |
| 7 | x8 | 0.066257 |
AdaBoost by hand Example¶
We considered $w_i$ and $\alpha_t = \frac12\log\frac{1-\varepsilon_t}{\varepsilon_t}$. Labels are $\{-1,1\}$.
x1 = np.arange(0.1, 1.01, 0.1)
x2 = np.array([0.5, 0.3, 0.1, 0.6, 0.7, 0.8, 0.5, 0.7, 0.8, 0.2])
y = np.array([1, 1, -1, -1, 1, 1, -1, 1, -1, -1])
X10 = np.column_stack([x1, x2])
xx, yy = np.meshgrid(np.linspace(0, 1.1, 220), np.linspace(0, 0.9, 180))
grid = np.column_stack([xx.ravel(), yy.ravel()])
def show_points(ax, weights=None):
s = 80 if weights is None else 4000 * np.asarray(weights)
ax.scatter(
X10[:, 0], X10[:, 1],
c=np.where(y > 0, "darkblue", "orange"),
marker="o", s=s,
linewidths=2,
edgecolors=np.where(y > 0, "darkblue", "orange"),
)
pos = y > 0
ax.scatter(X10[pos, 0], X10[pos, 1], marker="x", s=s[pos] if weights is not None else 80,
c="darkblue", linewidths=2)
ax.set_xlim(0, 1.1)
ax.set_ylim(0, 0.9)
fig, ax = plt.subplots()
show_points(ax)
ax.set_title("10-point AdaBoost toy data")
plt.show()
def stump_predict(grid, kind):
g = np.asarray(grid)
if kind == "x1_lt_0.25":
return np.where(g[:, 0] < 0.25, 1, -1)
if kind == "x2_gt_0.65":
return np.where(g[:, 1] > 0.65, 1, -1)
if kind == "x1_lt_0.85":
return np.where(g[:, 0] < 0.85, 1, -1)
raise ValueError(kind)
w = np.full(10, 1 / 10)
stumps = ["x1_lt_0.25", "x2_gt_0.65", "x1_lt_0.85"]
alphas = []
weight_hist = [w.copy()]
for kind in stumps:
pred = stump_predict(X10, kind)
err = np.sum(w * (pred != y))
err = np.clip(err, 1e-12, 1 - 1e-12)
a = 0.5 * np.log((1 - err) / err)
alphas.append(a)
w = w * np.exp(-a * y * pred)
w = w / w.sum()
weight_hist.append(w.copy())
print(f"{kind:16s} eps={err:.3f} alpha={a:.4f}")
print("F3(x) = " + " + ".join(f"{a:.4f} f{i+1}(x)" for i, a in enumerate(alphas)))
x1_lt_0.25 eps=0.300 alpha=0.4236 x2_gt_0.65 eps=0.214 alpha=0.6496 x1_lt_0.85 eps=0.136 alpha=0.9229 F3(x) = 0.4236 f1(x) + 0.6496 f2(x) + 0.9229 f3(x)
fig, axes = plt.subplots(2, 3, figsize=(12, 7))
panels = [
(axes[0, 0], "f1: x1 < 0.25", "x1_lt_0.25", weight_hist[0], False),
(axes[0, 1], "weights after f1", None, weight_hist[1], False),
(axes[0, 2], "f2: x2 > 0.65", "x2_gt_0.65", weight_hist[1], False),
(axes[1, 0], "weights after f2", None, weight_hist[2], False),
(axes[1, 1], "f3: x1 < 0.85", "x1_lt_0.85", weight_hist[2], False),
(axes[1, 2], "sign(F3)", "final", weight_hist[0], True),
]
for ax, title, kind, wcur, is_final in panels:
if kind == "final":
F = (alphas[0] * stump_predict(grid, stumps[0])
+ alphas[1] * stump_predict(grid, stumps[1])
+ alphas[2] * stump_predict(grid, stumps[2]))
ax.scatter(grid[:, 0], grid[:, 1], c=np.where(F > 0, "darkblue", "orange"),
s=1, alpha=0.35)
ax.axvline(0.25, color="black", lw=1)
ax.axhline(0.65, color="black", lw=1)
ax.axvline(0.85, color="black", lw=1)
show_points(ax)
elif kind is not None:
pred = stump_predict(grid, kind)
ax.scatter(grid[:, 0], grid[:, 1], c=np.where(pred > 0, "darkblue", "orange"),
s=1, alpha=0.35)
show_points(ax)
else:
show_points(ax, weights=wcur)
ax.set_title(title)
ax.set_xlim(0, 1.1)
ax.set_ylim(0, 0.9)
plt.tight_layout()
plt.show()
Same 10 points with sklearn.ensemble.AdaBoostClassifier (stumps, 3 rounds). Split locations need not match the hand calculation; the algorithm is the same.
ada = AdaBoostClassifier(
estimator=DecisionTreeClassifier(max_depth=1, min_samples_leaf=1),
n_estimators=3,
learning_rate=1.0,
random_state=0,
)
# sklearn wants {0,1} here
ada.fit(X10, (y == 1).astype(int))
print("estimator weights (alpha-like):", np.round(ada.estimator_weights_, 4))
print("estimator errors:", np.round(ada.estimator_errors_, 4))
estimator weights (alpha-like): [0.8473 1.2993 1.8458] estimator errors: [0.3 0.2143 0.1364]
AdaBoost / gradient boosting on a 1-d probability curve¶
R used gbm(..., distribution="adaboost") and plotted $1/(1+e^{-2F})$ at selected iteration counts.
rng = np.random.default_rng(1)
n = 1000
x = np.column_stack([np.linspace(0, 1, n), rng.uniform(0, 1, n)])
py = (np.sin(4 * np.pi * x[:, 0]) + 1) / 2
y1d = rng.binomial(1, py)
plt.scatter(x[:, 0], y1d + rng.uniform(-0.05, 0.05, n),
s=8, c=np.where(y1d == 1, "orange", "darkblue"))
plt.plot(x[:, 0], py, color="black", lw=2)
plt.ylim(-0.05, 1.05)
plt.xlabel("x")
plt.ylabel("P(Y = 1 | X = x)")
plt.title("1-d classification example")
plt.show()
ada1d = AdaBoostClassifier(
estimator=DecisionTreeClassifier(max_depth=1, min_samples_leaf=2),
n_estimators=200,
learning_rate=1.0,
random_state=1,
)
ada1d.fit(x, y1d)
AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1,
min_samples_leaf=2),
n_estimators=200, random_state=1)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
Fitted attributes
DecisionTreeClassifier(max_depth=1, min_samples_leaf=2)
Parameters
sizes = [1, 5, 10, 20, 50, 100]
fig, axes = plt.subplots(2, 3, figsize=(12, 7), sharex=True, sharey=True)
for ax, m in zip(axes.ravel(), sizes):
# Use the first m staged probability estimates
proba = None
for i, pr in enumerate(ada1d.staged_predict_proba(x), start=1):
if i == m:
proba = pr[:, 1]
break
ax.plot(x[:, 0], py, color="gray", lw=2)
ax.scatter(x[:, 0], y1d + rng.uniform(-0.05, 0.05, n),
s=6, c=np.where(y1d == 1, "orange", "darkblue"))
ax.plot(x[:, 0], proba, color="black", lw=1)
ax.set_title(f"# of iterations = {m}")
ax.set_ylim(-0.05, 1.05)
plt.tight_layout()
plt.show()
GradientBoostingClassifier is the closer analogue of gbm with distribution = "bernoulli". staged_predict_proba plays the role of predict(..., n.trees = k). There is no gbm.perf; pick (T) with a validation curve or validation_fraction + n_iter_no_change.
gb = GradientBoostingClassifier(
loss="log_loss",
n_estimators=200,
learning_rate=1.0, # large, as in the R notes, to make stages visible
min_samples_leaf=2,
subsample=0.8,
validation_fraction=0.2,
n_iter_no_change=20,
random_state=1,
)
gb.fit(x, y1d)
print("trees used (early stopping may cut this):", gb.n_estimators_)
trees used (early stopping may cut this): 21
xgboost¶
Same regression illustration as the R chunk. This cell uses xgboost when it loads, and sklearn otherwise.
For classification with xgboost set objective to binary:logistic or multi:softprob.
rng = np.random.default_rng(2025)
n, p = 2000, 15
X = rng.normal(size=(n, p))
def f_true(X):
return 2 * np.sin(X[:, 0]) + 0.5 * X[:, 1]**2 - 1.5 * (X[:, 2] > 0)
y = f_true(X) + rng.normal(scale=1.0, size=n)
idx = rng.choice(n, size=int(0.7 * n), replace=False)
mask = np.zeros(n, dtype=bool)
mask[idx] = True
if HAS_XGBOOST:
dtrain = xgb.DMatrix(X[mask], label=y[mask], feature_names=[f"x{i+1}" for i in range(p)])
dvalid = xgb.DMatrix(X[~mask], label=y[~mask], feature_names=[f"x{i+1}" for i in range(p)])
params = {
"objective": "reg:squarederror",
"eval_metric": "rmse",
"eta": 0.05,
"max_depth": 4,
"subsample": 0.8,
"colsample_bytree": 0.8,
"min_child_weight": 5,
"lambda": 1.0,
"alpha": 0.0,
}
fit = xgb.train(
params=params,
dtrain=dtrain,
num_boost_round=3000,
evals=[(dtrain, "train"), (dvalid, "valid")],
early_stopping_rounds=50,
verbose_eval=False,
)
print("Backend: xgboost")
print("Best iteration:", fit.best_iteration, " Valid RMSE:", round(fit.best_score, 4))
imp = fit.get_score(importance_type="gain")
imp_df = (pd.DataFrame({"feature": list(imp), "gain": list(imp.values())})
.sort_values("gain", ascending=False))
print(imp_df.head(8))
pred = fit.predict(dvalid)
else:
gbr = GradientBoostingRegressor(
n_estimators=400,
learning_rate=0.05,
max_depth=4,
subsample=0.8,
min_samples_leaf=5,
random_state=2025,
)
gbr.fit(X[mask], y[mask])
pred = gbr.predict(X[~mask])
print("Backend: sklearn GradientBoostingRegressor (xgboost/OpenMP unavailable)")
imp_df = (pd.DataFrame({
"feature": [f"x{i+1}" for i in range(p)],
"gain": gbr.feature_importances_,
}).sort_values("gain", ascending=False))
print(imp_df.head(8))
rmse = np.sqrt(np.mean((pred - y[~mask])**2))
print("RMSE:", round(rmse, 4))
Backend: sklearn GradientBoostingRegressor (xgboost/OpenMP unavailable) feature gain 0 x1 0.469687 2 x3 0.168983 1 x2 0.165296 7 x8 0.020251 4 x5 0.019769 14 x15 0.017333 5 x6 0.016234 12 x13 0.016091 RMSE: 1.0656
Controlled Comparison: CART, Bagging, Forest, Boosting¶
Same circular Bayes rule, one training draw, one test draw. R used cp from xerror, nbagg=200, mtry=1, and gbm with 5-fold CV. Here $T$ for boosting uses validation_fraction + early stopping.
def sim_circle(n, rng):
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
p = np.where(x1**2 + x2**2 < 0.6, 0.9, 0.1)
y = rng.binomial(1, p)
return np.column_stack([x1, x2]), y
rng = np.random.default_rng(10)
n = 1000
Xtr, ytr = sim_circle(n, rng)
Xte, yte = sim_circle(n, rng)
bayes_te = (Xte[:, 0]**2 + Xte[:, 1]**2 < 0.6).astype(int)
# 1. Pruned CART: choose ccp_alpha by a small validation split (stand-in for rpart xerror)
X_in, X_val, y_in, y_val = train_test_split(Xtr, ytr, test_size=0.3, random_state=10)
base = DecisionTreeClassifier(random_state=10)
path = base.cost_complexity_pruning_path(X_in, y_in)
best_alpha, best_err = 0.0, 1.0
for a in path.ccp_alphas:
t = DecisionTreeClassifier(random_state=10, ccp_alpha=a)
t.fit(X_in, y_in)
err = 1 - accuracy_score(y_val, t.predict(X_val))
if err <= best_err:
best_err, best_alpha = err, a
cart = DecisionTreeClassifier(random_state=10, ccp_alpha=best_alpha).fit(Xtr, ytr)
# 2. Bagging
bag = BaggingClassifier(
estimator=DecisionTreeClassifier(),
n_estimators=200,
random_state=10,
).fit(Xtr, ytr)
# 3. Forest, classification default max_features = sqrt(p) = 1 when p = 2
rf = RandomForestClassifier(
n_estimators=500,
max_features=1,
oob_score=True,
random_state=10,
n_jobs=-1,
).fit(Xtr, ytr)
# 4. Gradient boosting; T from internal validation
boost = GradientBoostingClassifier(
loss="log_loss",
n_estimators=400,
learning_rate=0.05,
max_depth=2, # interaction.depth = 2
subsample=0.8,
validation_fraction=0.2,
n_iter_no_change=20,
random_state=10,
).fit(Xtr, ytr)
print("CART ccp_alpha:", round(best_alpha, 5))
print("Forest OOB error:", round(1 - rf.oob_score_, 4))
print("Boosting trees used:", boost.n_estimators_)
CART ccp_alpha: 0.00408 Forest OOB error: 0.119 Boosting trees used: 180
models = {
"CART": cart,
"Bagging": bag,
"Random forest": rf,
f"Boosting ({boost.n_estimators_} trees)": boost,
}
fig, axes = plt.subplots(2, 2, figsize=(8, 8))
for ax, (name, model) in zip(axes.ravel(), models.items()):
decision_boundary(ax, model.predict, (-1, 1), (-1, 1), n=201)
ax.scatter(Xtr[:, 0], Xtr[:, 1],
c=np.where(ytr == 1, "darkblue", "orange"),
s=8, edgecolors="none")
ax.add_patch(plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2, color="black"))
ax.set_title(name)
ax.set_xticks([])
ax.set_yticks([])
ax.set_aspect("equal", "box")
plt.tight_layout()
plt.show()
maj = int(np.bincount(ytr).argmax())
preds = {
"Majority": np.full(len(yte), maj),
"CART": cart.predict(Xte),
"Bagging": bag.predict(Xte),
"Forest": rf.predict(Xte),
"Boosting": boost.predict(Xte),
}
rows = []
for name, ph in preds.items():
rows.append({
"method": name,
"test_error": round(np.mean(ph != yte), 3),
"agree_with_Bayes": round(np.mean(ph == bayes_te), 3),
})
pd.DataFrame(rows)
| method | test_error | agree_with_Bayes | |
|---|---|---|---|
| 0 | Majority | 0.496 | 0.507 |
| 1 | CART | 0.126 | 0.949 |
| 2 | Bagging | 0.123 | 0.954 |
| 3 | Forest | 0.119 | 0.962 |
| 4 | Boosting | 0.133 | 0.948 |
Notes when something does not match R exactly¶
rpartgrows with Gini (default) and prunes withcp; sklearn grows then prunes withccp_alpha. Trees will not be identical node-for-node.randomForestsamples with replacement of sizesampsize. sklearn'smax_samplessamples without replacement unless you write a custom bootstrap. OOB error is still valid whenoob_score=Trueand bootstrap is left at the defaultTrue.gbmwithdistribution="adaboost"is exponential loss.AdaBoostClassifieris the usual Python stand-in;GradientBoostingClassifier(loss="log_loss")is the stand-in fordistribution="bernoulli".pretty.gbm.treehas no one-line analogue; useplot_treeonada.estimators_[t]orboost.estimators_[t, 0].xgboostAPIs in R and Python are the same library; parameter names above match the R chunk.