Classification Trees & Ensemble Methods (Python Code)¶

In this notebook we present the Python code for all the R chunks in this Chapter.

Setup¶

In [48]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from pathlib import Path
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text
from sklearn.ensemble import (
    BaggingClassifier,
    RandomForestClassifier,
    AdaBoostClassifier,
    GradientBoostingClassifier,
    GradientBoostingRegressor,
)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# xgboost needs an OpenMP runtime (libomp / libgomp). If you do not have thin then skip it.
try:
    import xgboost as xgb
    HAS_XGBOOST = True
except Exception:
    xgb = None
    HAS_XGBOOST = False

plt.rcParams.update({
    "figure.figsize": (6, 4.5),
    "axes.grid": False,
})
In [49]:
def decision_boundary(ax, predict_fn, xlim, ylim, n=201, colors=("darkorange", "darkblue")):
    """Shade predicted class on a 2-d grid and draw the 0.5 contour."""
    xx, yy = np.meshgrid(
        np.linspace(*xlim, n),
        np.linspace(*ylim, n),
    )
    grid = np.column_stack([xx.ravel(), yy.ravel()])
    zz = np.asarray(predict_fn(grid)).reshape(xx.shape)
    ax.contourf(xx, yy, zz, levels=[-0.5, 0.5, 1.5],
                colors=colors, alpha=0.18)
    ax.contour(xx, yy, zz, levels=[0.5], colors="black", linewidths=1)
    return xx, yy, zz

Titanic Tree¶

We use the same Titanic data as in the R example:

In [50]:
DATA = Path("train.csv") 

cols = ["Survived", "Pclass", "Sex", "Age", "SibSp", "Parch"]
titanic = pd.read_csv(DATA, usecols=cols)
titanic["Sex"] = (titanic["Sex"] == "male").astype(int)  # rpart dummy-codes this internally
titanic_cc = titanic.dropna(subset=cols).copy()

X_all = titanic_cc[["Pclass", "Sex", "Age", "SibSp", "Parch"]]
y_all = titanic_cc["Survived"].astype(int)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_all, y_all, test_size=0.30, random_state=10, stratify=y_all
)

# Full-data tree (the rpart.plot figure)
tree_full = DecisionTreeClassifier(random_state=1, ccp_alpha=0.02)
tree_full.fit(X_all, y_all)

fig, ax = plt.subplots(figsize=(12, 7))
plot_tree(
    tree_full,
    feature_names=list(X_all.columns),
    class_names=["died", "survived"],
    filled=True,
    impurity=True,
    proportion=True,
    ax=ax,
)
ax.set_title("Pruned classification tree (ccp_alpha = 0.02)")
plt.tight_layout()
plt.show()

print(export_text(tree_full, feature_names=list(X_all.columns)))
No description has been provided for this image
|--- Sex <= 0.50
|   |--- Pclass <= 2.50
|   |   |--- class: 1
|   |--- Pclass >  2.50
|   |   |--- class: 0
|--- Sex >  0.50
|   |--- class: 0

In [51]:
maj = int(y_tr.value_counts().idxmax())
base_err = np.mean(y_te.to_numpy() != maj)

tree_hold = DecisionTreeClassifier(random_state=1, ccp_alpha=0.02)
tree_hold.fit(X_tr, y_tr)
tree_err_tr = 1 - accuracy_score(y_tr, tree_hold.predict(X_tr))
tree_err_te = 1 - accuracy_score(y_te, tree_hold.predict(X_te))

logit = LogisticRegression(max_iter=1000)
logit.fit(X_tr, y_tr)
glm_err_te = 1 - accuracy_score(y_te, logit.predict(X_te))

pd.DataFrame({
    "method": [
        "Majority class (holdout)",
        "Pruned tree (train)",
        "Pruned tree (holdout)",
        "Logistic regression (holdout)",
    ],
    "misclassification": np.round(
        [base_err, tree_err_tr, tree_err_te, glm_err_te], 3
    ),
})
Out[51]:
method misclassification
0 Majority class (holdout) 0.405
1 Pruned tree (train) 0.194
2 Pruned tree (holdout) 0.219
3 Logistic regression (holdout) 0.214

Circle Simulation and a Single Tree¶

In [52]:
rng = np.random.default_rng(1)
n = 500
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
p = np.where(x1**2 + x2**2 < 0.6, 0.9, 0.1)
y = rng.binomial(1, p)

fig, ax = plt.subplots()
ax.scatter(x1, x2, c=np.where(y == 1, "darkblue", "darkorange"), s=18, edgecolors="none")
circ = plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2)
ax.add_patch(circ)
ax.set_aspect("equal", "box")
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.set_title("Circle simulation")
plt.show()

X_circ = np.column_stack([x1, x2])
tree_circ = DecisionTreeClassifier(random_state=1)
tree_circ.fit(X_circ, y)

fig, ax = plt.subplots(figsize=(10, 6))
plot_tree(tree_circ, feature_names=["x1", "x2"], class_names=["0", "1"],
          filled=True, max_depth=4, ax=ax)
ax.set_title("Unpruned tree (depth capped in the plot only)")
plt.tight_layout()
plt.show()
No description has been provided for this image
No description has been provided for this image

Cost-complexity path (R: rpart.fit$cptable). Then prune at a chosen ccp_alpha. Remark: Do not copy a cp value from the R notes into ccp_alpha. ccp_alpha is the same idea as rpart's cp (cost-complexity pruning)

In [53]:
path = tree_circ.cost_complexity_pruning_path(X_circ, y)
cptable = pd.DataFrame({
    "ccp_alpha": path.ccp_alphas,
    "impurity_leaves": path.impurities,
})
cptable
Out[53]:
ccp_alpha impurity_leaves
0 0.000000 0.000000
1 0.001500 0.003000
2 0.001714 0.006429
3 0.001771 0.020600
4 0.001786 0.024171
5 0.001810 0.031410
6 0.001818 0.038682
7 0.001846 0.042375
8 0.001857 0.046089
9 0.001857 0.053517
10 0.001867 0.060985
11 0.001882 0.064750
12 0.001897 0.072337
13 0.001900 0.076137
14 0.001902 0.083747
15 0.002000 0.089747
16 0.002012 0.093771
17 0.002037 0.097846
18 0.002390 0.112187
19 0.002400 0.116987
20 0.002400 0.121787
21 0.002400 0.124187
22 0.002497 0.129180
23 0.002667 0.134514
24 0.002829 0.143000
25 0.003126 0.146125
26 0.003200 0.149325
27 0.003333 0.152659
28 0.003500 0.156159
29 0.003579 0.163317
30 0.003955 0.171226
31 0.004345 0.179915
32 0.005352 0.201322
33 0.006199 0.219918
34 0.006341 0.226258
35 0.006824 0.239907
36 0.007088 0.246995
37 0.010683 0.268361
38 0.016029 0.284389
39 0.053703 0.499200
In [54]:
# Pick an alpha near the R notes' idea of a small prune (not the same number as cp = 0.041)
alpha = path.ccp_alphas[max(0, len(path.ccp_alphas) // 3)]
pruned = DecisionTreeClassifier(random_state=1, ccp_alpha=alpha)
pruned.fit(X_circ, y)

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
plot_tree(pruned, feature_names=["x1", "x2"], class_names=["0", "1"],
          filled=True, ax=axes[0])
axes[0].set_title(f"Pruned tree (ccp_alpha = {alpha:.4f})")

decision_boundary(axes[1], pruned.predict, (-1, 1), (-1, 1))
axes[1].scatter(x1, x2, c=np.where(y == 1, "darkblue", "darkorange"),
                s=16, edgecolors="none")
axes[1].add_patch(plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2))
axes[1].set_aspect("equal", "box")
axes[1].set_title("Axis-aligned staircase vs Bayes circle")
plt.tight_layout()
plt.show()
No description has been provided for this image

Impurity Measures¶

R defined gini, shannon, error, and the split gain score.

In [55]:
def _freqs(y):
    y = np.asarray(y)
    _, counts = np.unique(y, return_counts=True)
    return counts / counts.sum()

def gini(y):
    p = _freqs(y)
    return float(np.sum(p * (1 - p)))

def shannon(y):
    p = _freqs(y)
    p = p[p > 0]
    return float(-np.sum(p * np.log(p)))

def misclass(y):
    p = _freqs(y)
    return float(1 - p.max())

def split_gain(left, right, measure):
    left, right = np.asarray(left), np.asarray(right)
    parent = np.concatenate([left, right])
    n_l, n_r = len(left), len(right)
    n = n_l + n_r
    return measure(parent) - (n_l / n) * measure(left) - (n_r / n) * measure(right)

TL = np.repeat(1, 3)
TR = np.array([1, 1, 1, 1, 0, 0, 0])
print("Gini gain     ", split_gain(TL, TR, gini))
print("Entropy gain  ", split_gain(TL, TR, shannon))
print("Error gain    ", split_gain(TL, TR, misclass))
Gini gain      0.07714285714285724
Entropy gain   0.13282862876456336
Error gain     5.551115123125783e-17
In [56]:
p = np.linspace(0, 1, 201)
g = 2 * p * (1 - p)
s = np.zeros_like(p)
mid = (p > 0) & (p < 1)
s[mid] = -p[mid] * np.log(p[mid]) - (1 - p[mid]) * np.log(1 - p[mid])
e = np.minimum(p, 1 - p)

plt.plot(p, s, lw=2, label="Entropy")
plt.plot(p, g, lw=2, label="Gini")
plt.plot(p, e, lw=2, label="Error")
plt.xlabel("p")
plt.ylabel("Impurity")
plt.legend()
plt.title("Binary impurity curves")
plt.show()
No description has been provided for this image

Bagging vs One Tree (diagonal strip)¶

R: rpart vs ipred::bagging(..., nbagg = 200).

In [57]:
rng = np.random.default_rng(2)
n = 1000
x1 = rng.uniform(-1, 1, n)
x2 = rng.uniform(-1, 1, n)
inside = (x1 + x2 > -0.5) & (x1 + x2 < 0.5)
y = rng.binomial(1, np.where(inside, 0.8, 0.2))
X = np.column_stack([x1, x2])

cart = DecisionTreeClassifier(random_state=2)
cart.fit(X, y)

bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=200,
    random_state=2,
)
bag.fit(X, y)

fig, axes = plt.subplots(1, 2, figsize=(10, 4.5))
for ax, model, title in [(axes[0], cart, "CART"), (axes[1], bag, "Bagging")]:
    decision_boundary(ax, model.predict, (-1, 1), (-1, 1))
    ax.scatter(x1, x2, c=np.where(y == 1, "darkblue", "orange"),
               s=12, edgecolors="none")
    ax.set_title(title)
    ax.set_xticks([])
    ax.set_yticks([])
plt.tight_layout()
plt.show()
No description has been provided for this image

Random forest, OOB, Bayes agreement, importance¶

R: randomForest(X, y, ntree=1000, mtry=7, nodesize=10, sampsize=800).

sklearn names:

  • mtry → max_features
  • nodesize → min_samples_leaf
  • sampsize → max_samples
  • OOB error → 1 - oob_score_
In [58]:
rng = np.random.default_rng(542)
n, p = 1000, 10
X = rng.uniform(-1, 1, size=(n, p))
inside = (X[:, 0] + X[:, 1] > -0.5) & (X[:, 0] + X[:, 1] < 0.5)
y = rng.binomial(1, np.where(inside, 0.8, 0.2))

rf = RandomForestClassifier(
    n_estimators=1000,
    max_features=7,       # mtry
    min_samples_leaf=10,  # nodesize
    max_samples=800,      # sampsize
    oob_score=True,
    random_state=542,
    n_jobs=-1,
)
rf.fit(X, y)
print("OOB error:", round(1 - rf.oob_score_, 4))

Xtest = rng.uniform(-1, 1, size=(n, p))
bayes = ((Xtest[:, 0] + Xtest[:, 1] > -0.5) &
         (Xtest[:, 0] + Xtest[:, 1] < 0.5)).astype(int)
agree = np.mean(rf.predict(Xtest) == bayes)
print("Agreement with Bayes rule:", round(agree, 4))

imp = pd.DataFrame({
    "feature": [f"x{j+1}" for j in range(p)],
    "gini_importance": rf.feature_importances_,
}).sort_values("gini_importance", ascending=False)
imp
OOB error: 0.301
Agreement with Bayes rule: 0.847
Out[58]:
feature gini_importance
1 x2 0.204473
0 x1 0.192906
6 x7 0.082063
2 x3 0.081233
8 x9 0.080029
5 x6 0.075407
4 x5 0.074082
9 x10 0.072909
3 x4 0.070642
7 x8 0.066257

AdaBoost by hand Example¶

We considered $w_i$ and $\alpha_t = \frac12\log\frac{1-\varepsilon_t}{\varepsilon_t}$. Labels are $\{-1,1\}$.

In [59]:
x1 = np.arange(0.1, 1.01, 0.1)
x2 = np.array([0.5, 0.3, 0.1, 0.6, 0.7, 0.8, 0.5, 0.7, 0.8, 0.2])
y = np.array([1, 1, -1, -1, 1, 1, -1, 1, -1, -1])
X10 = np.column_stack([x1, x2])

xx, yy = np.meshgrid(np.linspace(0, 1.1, 220), np.linspace(0, 0.9, 180))
grid = np.column_stack([xx.ravel(), yy.ravel()])

def show_points(ax, weights=None):
    s = 80 if weights is None else 4000 * np.asarray(weights)
    ax.scatter(
        X10[:, 0], X10[:, 1],
        c=np.where(y > 0, "darkblue", "orange"),
        marker="o", s=s,
        linewidths=2,
        edgecolors=np.where(y > 0, "darkblue", "orange"),
    )
    pos = y > 0
    ax.scatter(X10[pos, 0], X10[pos, 1], marker="x", s=s[pos] if weights is not None else 80,
               c="darkblue", linewidths=2)
    ax.set_xlim(0, 1.1)
    ax.set_ylim(0, 0.9)

fig, ax = plt.subplots()
show_points(ax)
ax.set_title("10-point AdaBoost toy data")
plt.show()
No description has been provided for this image
In [60]:
def stump_predict(grid, kind):
    g = np.asarray(grid)
    if kind == "x1_lt_0.25":
        return np.where(g[:, 0] < 0.25, 1, -1)
    if kind == "x2_gt_0.65":
        return np.where(g[:, 1] > 0.65, 1, -1)
    if kind == "x1_lt_0.85":
        return np.where(g[:, 0] < 0.85, 1, -1)
    raise ValueError(kind)

w = np.full(10, 1 / 10)
stumps = ["x1_lt_0.25", "x2_gt_0.65", "x1_lt_0.85"]
alphas = []
weight_hist = [w.copy()]

for kind in stumps:
    pred = stump_predict(X10, kind)
    err = np.sum(w * (pred != y))
    err = np.clip(err, 1e-12, 1 - 1e-12)
    a = 0.5 * np.log((1 - err) / err)
    alphas.append(a)
    w = w * np.exp(-a * y * pred)
    w = w / w.sum()
    weight_hist.append(w.copy())
    print(f"{kind:16s}  eps={err:.3f}  alpha={a:.4f}")

print("F3(x) = " + " + ".join(f"{a:.4f} f{i+1}(x)" for i, a in enumerate(alphas)))
x1_lt_0.25        eps=0.300  alpha=0.4236
x2_gt_0.65        eps=0.214  alpha=0.6496
x1_lt_0.85        eps=0.136  alpha=0.9229
F3(x) = 0.4236 f1(x) + 0.6496 f2(x) + 0.9229 f3(x)
In [61]:
fig, axes = plt.subplots(2, 3, figsize=(12, 7))
panels = [
    (axes[0, 0], "f1: x1 < 0.25", "x1_lt_0.25", weight_hist[0], False),
    (axes[0, 1], "weights after f1", None, weight_hist[1], False),
    (axes[0, 2], "f2: x2 > 0.65", "x2_gt_0.65", weight_hist[1], False),
    (axes[1, 0], "weights after f2", None, weight_hist[2], False),
    (axes[1, 1], "f3: x1 < 0.85", "x1_lt_0.85", weight_hist[2], False),
    (axes[1, 2], "sign(F3)", "final", weight_hist[0], True),
]

for ax, title, kind, wcur, is_final in panels:
    if kind == "final":
        F = (alphas[0] * stump_predict(grid, stumps[0])
             + alphas[1] * stump_predict(grid, stumps[1])
             + alphas[2] * stump_predict(grid, stumps[2]))
        ax.scatter(grid[:, 0], grid[:, 1], c=np.where(F > 0, "darkblue", "orange"),
                   s=1, alpha=0.35)
        ax.axvline(0.25, color="black", lw=1)
        ax.axhline(0.65, color="black", lw=1)
        ax.axvline(0.85, color="black", lw=1)
        show_points(ax)
    elif kind is not None:
        pred = stump_predict(grid, kind)
        ax.scatter(grid[:, 0], grid[:, 1], c=np.where(pred > 0, "darkblue", "orange"),
                   s=1, alpha=0.35)
        show_points(ax)
    else:
        show_points(ax, weights=wcur)
    ax.set_title(title)
    ax.set_xlim(0, 1.1)
    ax.set_ylim(0, 0.9)
plt.tight_layout()
plt.show()
No description has been provided for this image

Same 10 points with sklearn.ensemble.AdaBoostClassifier (stumps, 3 rounds). Split locations need not match the hand calculation; the algorithm is the same.

In [62]:
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, min_samples_leaf=1),
    n_estimators=3,
    learning_rate=1.0,
    random_state=0,
)
# sklearn wants {0,1} here
ada.fit(X10, (y == 1).astype(int))
print("estimator weights (alpha-like):", np.round(ada.estimator_weights_, 4))
print("estimator errors:", np.round(ada.estimator_errors_, 4))
estimator weights (alpha-like): [0.8473 1.2993 1.8458]
estimator errors: [0.3    0.2143 0.1364]

AdaBoost / gradient boosting on a 1-d probability curve¶

R used gbm(..., distribution="adaboost") and plotted $1/(1+e^{-2F})$ at selected iteration counts.

In [63]:
rng = np.random.default_rng(1)
n = 1000
x = np.column_stack([np.linspace(0, 1, n), rng.uniform(0, 1, n)])
py = (np.sin(4 * np.pi * x[:, 0]) + 1) / 2
y1d = rng.binomial(1, py)

plt.scatter(x[:, 0], y1d + rng.uniform(-0.05, 0.05, n),
            s=8, c=np.where(y1d == 1, "orange", "darkblue"))
plt.plot(x[:, 0], py, color="black", lw=2)
plt.ylim(-0.05, 1.05)
plt.xlabel("x")
plt.ylabel("P(Y = 1 | X = x)")
plt.title("1-d classification example")
plt.show()

ada1d = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1, min_samples_leaf=2),
    n_estimators=200,
    learning_rate=1.0,
    random_state=1,
)
ada1d.fit(x, y1d)
No description has been provided for this image
Out[63]:
AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1,
                                                    min_samples_leaf=2),
                   n_estimators=200, random_state=1)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
estimator estimator: object, default=None

The base estimator from which the boosted ensemble is built.
Support for sample weighting is required, as well as proper
``classes_`` and ``n_classes_`` attributes. If ``None``, then
the base estimator is :class:`~sklearn.tree.DecisionTreeClassifier`
initialized with `max_depth=1`.

.. versionadded:: 1.2
`base_estimator` was renamed to `estimator`.
DecisionTreeC...amples_leaf=2)
n_estimators n_estimators: int, default=50

The maximum number of estimators at which boosting is terminated.
In case of perfect fit, the learning procedure is stopped early.
Values must be in the range `[1, inf)`.
200
random_state random_state: int, RandomState instance or None, default=None

Controls the random seed given at each `estimator` at each
boosting iteration.
Thus, it is only used when `estimator` exposes a `random_state`.
Pass an int for reproducible output across multiple function calls.
See :term:`Glossary <random_state>`.
1
learning_rate learning_rate: float, default=1.0

Weight applied to each classifier at each boosting iteration. A higher
learning rate increases the contribution of each classifier. There is
a trade-off between the `learning_rate` and `n_estimators` parameters.
Values must be in the range `(0.0, inf)`.
1.0
Fitted attributes
Name Type Value
classes_ classes_: ndarray of shape (n_classes,)

The classes labels.
ndarray[int64](2,) [0,1]
estimator_ estimator_: estimator

The base estimator from which the ensemble is grown.

.. versionadded:: 1.2
`base_estimator_` was renamed to `estimator_`.
DecisionTreeClassifier DecisionTreeC...amples_leaf=2)
estimator_errors_ estimator_errors_: ndarray of floats

Classification error for each estimator in the boosted
ensemble.
ndarray[float64](200,) [0.33,0.33,0.32,...,0.48,0.48,0.48]
estimator_weights_ estimator_weights_: ndarray of floats

Weights for each estimator in the boosted ensemble.
ndarray[float64](200,) [0.7 ,0.72,0.77,...,0.07,0.08,0.08]
estimators_ estimators_: list of classifiers

The collection of fitted sub-estimators.
list [DecisionTreeC...te=1791095845), DecisionTreeC...te=2135392491), DecisionTreeC...ate=946286476), DecisionTreeC...te=1857819720), ...]
feature_importances_ feature_importances_: ndarray of shape (n_features,)

The impurity-based feature importances if supported by the
``estimator`` (when based on decision trees).

Warning: impurity-based feature importances can be misleading for
high cardinality features (many unique values). See
:func:`sklearn.inspection.permutation_importance` as an alternative.
ndarray[float64](2,) [0.84,0.16]
n_classes_ n_classes_: int

The number of classes.
int 2
n_features_in_ n_features_in_: int

Number of features seen during :term:`fit`.

.. versionadded:: 0.24
int 2
DecisionTreeClassifier(max_depth=1, min_samples_leaf=2)
Parameters
max_depth max_depth: int, default=None

The maximum depth of the tree. If None, then nodes are expanded until
all leaves are pure or until all leaves contain less than
min_samples_split samples.
1
min_samples_leaf min_samples_leaf: int or float, default=1

The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at
least ``min_samples_leaf`` training samples in each of the left and
right branches. This may have the effect of smoothing the model,
especially in regression.

- If int, then consider `min_samples_leaf` as the minimum number.
- If float, then `min_samples_leaf` is a fraction and
`ceil(min_samples_leaf * n_samples)` are the minimum
number of samples for each node.

.. versionchanged:: 0.18
Added float values for fractions.
2
criterion criterion: {"gini", "entropy", "log_loss"}, default="gini"

The function to measure the quality of a split. Supported criteria are
"gini" for the Gini impurity and "log_loss" and "entropy" both for the
Shannon information gain, see :ref:`tree_mathematical_formulation`.
'gini'
splitter splitter: {"best", "random"}, default="best"

The strategy used to choose the split at each node. Supported
strategies are "best" to choose the best split and "random" to choose
the best random split.
'best'
min_samples_split min_samples_split: int or float, default=2

The minimum number of samples required to split an internal node:

- If int, then consider `min_samples_split` as the minimum number.
- If float, then `min_samples_split` is a fraction and
`ceil(min_samples_split * n_samples)` are the minimum
number of samples for each split.

.. versionchanged:: 0.18
Added float values for fractions.
2
min_weight_fraction_leaf min_weight_fraction_leaf: float, default=0.0

The minimum weighted fraction of the sum total of weights (of all
the input samples) required to be at a leaf node. Samples have
equal weight when sample_weight is not provided.
0.0
max_features max_features: int, float or {"sqrt", "log2"}, default=None

The number of features to consider when looking for the best split:

- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
`max(1, int(max_features * n_features_in_))` features are considered at
each split.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
- If None, then `max_features=n_features`.

.. note::

The search for a split does not stop until at least one
valid partition of the node samples is found, even if it requires to
effectively inspect more than ``max_features`` features.
None
random_state random_state: int, RandomState instance or None, default=None

Controls the randomness of the estimator. The features are always
randomly permuted at each split, even if ``splitter`` is set to
``"best"``. When ``max_features < n_features``, the algorithm will
select ``max_features`` at random at each split before finding the best
split among them. But the best found split may vary across different
runs, even if ``max_features=n_features``. That is the case, if the
improvement of the criterion is identical for several splits and one
split has to be selected at random. To obtain a deterministic behaviour
during fitting, ``random_state`` has to be fixed to an integer.
See :term:`Glossary <random_state>` for details.
None
max_leaf_nodes max_leaf_nodes: int, default=None

Grow a tree with ``max_leaf_nodes`` in best-first fashion.
Best nodes are defined as relative reduction in impurity.
If None then unlimited number of leaf nodes.
None
min_impurity_decrease min_impurity_decrease: float, default=0.0

A node will be split if this split induces a decrease of the impurity
greater than or equal to this value.

The weighted impurity decrease equation is the following::

N_t / N * (impurity - N_t_R / N_t * right_impurity
- N_t_L / N_t * left_impurity)

where ``N`` is the total number of samples, ``N_t`` is the number of
samples at the current node, ``N_t_L`` is the number of samples in the
left child, and ``N_t_R`` is the number of samples in the right child.

``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
if ``sample_weight`` is passed.

.. versionadded:: 0.19
0.0
class_weight class_weight: dict, list of dict or "balanced", default=None

Weights associated with classes in the form ``{class_label: weight}``.
If None, all classes are supposed to have weight one. For
multi-output problems, a list of dicts can be provided in the same
order as the columns of y.

Note that for multioutput (including multilabel) weights should be
defined for each class of every column in its own dict. For example,
for four-class multilabel classification weights should be
[{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of
[{1:1}, {2:5}, {3:1}, {4:1}].

The "balanced" mode uses the values of y to automatically adjust
weights inversely proportional to class frequencies in the input data
as ``n_samples / (n_classes * np.bincount(y))``

For multi-output, the weights of each column of y will be multiplied.

Note that these weights will be multiplied with sample_weight (passed
through the fit method) if sample_weight is specified.
None
ccp_alpha ccp_alpha: non-negative float, default=0.0

Complexity parameter used for Minimal Cost-Complexity Pruning. The
subtree with the largest cost complexity that is smaller than
``ccp_alpha`` will be chosen. By default, no pruning is performed. See
:ref:`minimal_cost_complexity_pruning` for details. See
:ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py`
for an example of such pruning.

.. versionadded:: 0.22
0.0
monotonic_cst monotonic_cst: array-like of int of shape (n_features), default=None

Indicates the monotonicity constraint to enforce on each feature.
- 1: monotonic increase
- 0: no constraint
- -1: monotonic decrease

If monotonic_cst is None, no constraints are applied.

Monotonicity constraints are not supported for:
- multiclass classifications (i.e. when `n_classes > 2`),
- multioutput classifications (i.e. when `n_outputs_ > 1`).

The constraints hold over the probability of the positive class.

Read more in the :ref:`User Guide <monotonic_cst_gbdt>`.

.. versionadded:: 1.4
None
In [64]:
sizes = [1, 5, 10, 20, 50, 100]
fig, axes = plt.subplots(2, 3, figsize=(12, 7), sharex=True, sharey=True)
for ax, m in zip(axes.ravel(), sizes):
    # Use the first m staged probability estimates
    proba = None
    for i, pr in enumerate(ada1d.staged_predict_proba(x), start=1):
        if i == m:
            proba = pr[:, 1]
            break
    ax.plot(x[:, 0], py, color="gray", lw=2)
    ax.scatter(x[:, 0], y1d + rng.uniform(-0.05, 0.05, n),
               s=6, c=np.where(y1d == 1, "orange", "darkblue"))
    ax.plot(x[:, 0], proba, color="black", lw=1)
    ax.set_title(f"# of iterations = {m}")
    ax.set_ylim(-0.05, 1.05)
plt.tight_layout()
plt.show()
No description has been provided for this image

GradientBoostingClassifier is the closer analogue of gbm with distribution = "bernoulli". staged_predict_proba plays the role of predict(..., n.trees = k). There is no gbm.perf; pick (T) with a validation curve or validation_fraction + n_iter_no_change.

In [65]:
gb = GradientBoostingClassifier(
    loss="log_loss",
    n_estimators=200,
    learning_rate=1.0,   # large, as in the R notes, to make stages visible
    min_samples_leaf=2,
    subsample=0.8,
    validation_fraction=0.2,
    n_iter_no_change=20,
    random_state=1,
)
gb.fit(x, y1d)
print("trees used (early stopping may cut this):", gb.n_estimators_)
trees used (early stopping may cut this): 21

xgboost¶

Same regression illustration as the R chunk. This cell uses xgboost when it loads, and sklearn otherwise.

For classification with xgboost set objective to binary:logistic or multi:softprob.

In [66]:
rng = np.random.default_rng(2025)
n, p = 2000, 15
X = rng.normal(size=(n, p))

def f_true(X):
    return 2 * np.sin(X[:, 0]) + 0.5 * X[:, 1]**2 - 1.5 * (X[:, 2] > 0)

y = f_true(X) + rng.normal(scale=1.0, size=n)
idx = rng.choice(n, size=int(0.7 * n), replace=False)
mask = np.zeros(n, dtype=bool)
mask[idx] = True

if HAS_XGBOOST:
    dtrain = xgb.DMatrix(X[mask], label=y[mask], feature_names=[f"x{i+1}" for i in range(p)])
    dvalid = xgb.DMatrix(X[~mask], label=y[~mask], feature_names=[f"x{i+1}" for i in range(p)])
    params = {
        "objective": "reg:squarederror",
        "eval_metric": "rmse",
        "eta": 0.05,
        "max_depth": 4,
        "subsample": 0.8,
        "colsample_bytree": 0.8,
        "min_child_weight": 5,
        "lambda": 1.0,
        "alpha": 0.0,
    }
    fit = xgb.train(
        params=params,
        dtrain=dtrain,
        num_boost_round=3000,
        evals=[(dtrain, "train"), (dvalid, "valid")],
        early_stopping_rounds=50,
        verbose_eval=False,
    )
    print("Backend: xgboost")
    print("Best iteration:", fit.best_iteration, "  Valid RMSE:", round(fit.best_score, 4))
    imp = fit.get_score(importance_type="gain")
    imp_df = (pd.DataFrame({"feature": list(imp), "gain": list(imp.values())})
              .sort_values("gain", ascending=False))
    print(imp_df.head(8))
    pred = fit.predict(dvalid)
else:
    gbr = GradientBoostingRegressor(
        n_estimators=400,
        learning_rate=0.05,
        max_depth=4,
        subsample=0.8,
        min_samples_leaf=5,
        random_state=2025,
    )
    gbr.fit(X[mask], y[mask])
    pred = gbr.predict(X[~mask])
    print("Backend: sklearn GradientBoostingRegressor (xgboost/OpenMP unavailable)")
    imp_df = (pd.DataFrame({
        "feature": [f"x{i+1}" for i in range(p)],
        "gain": gbr.feature_importances_,
    }).sort_values("gain", ascending=False))
    print(imp_df.head(8))

rmse = np.sqrt(np.mean((pred - y[~mask])**2))
print("RMSE:", round(rmse, 4))
Backend: sklearn GradientBoostingRegressor (xgboost/OpenMP unavailable)
   feature      gain
0       x1  0.469687
2       x3  0.168983
1       x2  0.165296
7       x8  0.020251
4       x5  0.019769
14     x15  0.017333
5       x6  0.016234
12     x13  0.016091
RMSE: 1.0656

Controlled Comparison: CART, Bagging, Forest, Boosting¶

Same circular Bayes rule, one training draw, one test draw. R used cp from xerror, nbagg=200, mtry=1, and gbm with 5-fold CV. Here $T$ for boosting uses validation_fraction + early stopping.

In [67]:
def sim_circle(n, rng):
    x1 = rng.uniform(-1, 1, n)
    x2 = rng.uniform(-1, 1, n)
    p = np.where(x1**2 + x2**2 < 0.6, 0.9, 0.1)
    y = rng.binomial(1, p)
    return np.column_stack([x1, x2]), y

rng = np.random.default_rng(10)
n = 1000
Xtr, ytr = sim_circle(n, rng)
Xte, yte = sim_circle(n, rng)
bayes_te = (Xte[:, 0]**2 + Xte[:, 1]**2 < 0.6).astype(int)

# 1. Pruned CART: choose ccp_alpha by a small validation split (stand-in for rpart xerror)
X_in, X_val, y_in, y_val = train_test_split(Xtr, ytr, test_size=0.3, random_state=10)
base = DecisionTreeClassifier(random_state=10)
path = base.cost_complexity_pruning_path(X_in, y_in)
best_alpha, best_err = 0.0, 1.0
for a in path.ccp_alphas:
    t = DecisionTreeClassifier(random_state=10, ccp_alpha=a)
    t.fit(X_in, y_in)
    err = 1 - accuracy_score(y_val, t.predict(X_val))
    if err <= best_err:
        best_err, best_alpha = err, a
cart = DecisionTreeClassifier(random_state=10, ccp_alpha=best_alpha).fit(Xtr, ytr)

# 2. Bagging
bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=200,
    random_state=10,
).fit(Xtr, ytr)

# 3. Forest, classification default max_features = sqrt(p) = 1 when p = 2
rf = RandomForestClassifier(
    n_estimators=500,
    max_features=1,
    oob_score=True,
    random_state=10,
    n_jobs=-1,
).fit(Xtr, ytr)

# 4. Gradient boosting; T from internal validation
boost = GradientBoostingClassifier(
    loss="log_loss",
    n_estimators=400,
    learning_rate=0.05,
    max_depth=2,            # interaction.depth = 2
    subsample=0.8,
    validation_fraction=0.2,
    n_iter_no_change=20,
    random_state=10,
).fit(Xtr, ytr)

print("CART ccp_alpha:", round(best_alpha, 5))
print("Forest OOB error:", round(1 - rf.oob_score_, 4))
print("Boosting trees used:", boost.n_estimators_)
CART ccp_alpha: 0.00408
Forest OOB error: 0.119
Boosting trees used: 180
In [68]:
models = {
    "CART": cart,
    "Bagging": bag,
    "Random forest": rf,
    f"Boosting ({boost.n_estimators_} trees)": boost,
}

fig, axes = plt.subplots(2, 2, figsize=(8, 8))
for ax, (name, model) in zip(axes.ravel(), models.items()):
    decision_boundary(ax, model.predict, (-1, 1), (-1, 1), n=201)
    ax.scatter(Xtr[:, 0], Xtr[:, 1],
               c=np.where(ytr == 1, "darkblue", "orange"),
               s=8, edgecolors="none")
    ax.add_patch(plt.Circle((0, 0), np.sqrt(0.6), fill=False, lw=2, color="black"))
    ax.set_title(name)
    ax.set_xticks([])
    ax.set_yticks([])
    ax.set_aspect("equal", "box")
plt.tight_layout()
plt.show()
No description has been provided for this image
In [69]:
maj = int(np.bincount(ytr).argmax())
preds = {
    "Majority": np.full(len(yte), maj),
    "CART": cart.predict(Xte),
    "Bagging": bag.predict(Xte),
    "Forest": rf.predict(Xte),
    "Boosting": boost.predict(Xte),
}
rows = []
for name, ph in preds.items():
    rows.append({
        "method": name,
        "test_error": round(np.mean(ph != yte), 3),
        "agree_with_Bayes": round(np.mean(ph == bayes_te), 3),
    })
pd.DataFrame(rows)
Out[69]:
method test_error agree_with_Bayes
0 Majority 0.496 0.507
1 CART 0.126 0.949
2 Bagging 0.123 0.954
3 Forest 0.119 0.962
4 Boosting 0.133 0.948

Notes when something does not match R exactly¶

  • rpart grows with Gini (default) and prunes with cp; sklearn grows then prunes with ccp_alpha. Trees will not be identical node-for-node.
  • randomForest samples with replacement of size sampsize. sklearn's max_samples samples without replacement unless you write a custom bootstrap. OOB error is still valid when oob_score=True and bootstrap is left at the default True.
  • gbm with distribution="adaboost" is exponential loss. AdaBoostClassifier is the usual Python stand-in; GradientBoostingClassifier(loss="log_loss") is the stand-in for distribution="bernoulli".
  • pretty.gbm.tree has no one-line analogue; use plot_tree on ada.estimators_[t] or boost.estimators_[t, 0].
  • xgboost APIs in R and Python are the same library; parameter names above match the R chunk.