6.3b Tree-Based Regression Models in Python¶

This notebook is the Python counterpart of the R section Tree-Based Regression Models in R. The analysis is the same -- white-wine quality, single trees, cost-complexity pruning, random forests, and gradient boosting -- implemented with scikit-learn, pandas, and matplotlib.

sklearn does not reproduce tree, rpart, randomForest, or gbm line for line. Differences that change the output (a number or a plot) are discussed.

Two R packages fit trees in the notes: tree and rpart. In Python, we have one class, DecisionTreeRegressor that covers both illustration and pruning. Ensembles are RandomForestRegressor and GradientBoostingRegressor.

In [25]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeRegressor, plot_tree
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.inspection import permutation_importance
from sklearn.metrics import mean_squared_error

plt.style.use("seaborn-v0_8-darkgrid")
RANDOM_STATE = 598  # same seed as the R notes

The whitewines.csv data set contains information on white variants of the Portuguese "Vinho Verde" wine.

The columns are: (a) fixed acidity, (b) volatile acidity, (c) citric acid, (d) residual sugar, (e) chlorides, (f) free sulfur dioxide, (g) total sulfur dioxide, (h) density, (i) pH, (j) sulphates, (k) alcohol, (l) quality (score between 0 and 10).

The file used here is the same semicolon-separated table as in the R notes and can be downloaded from there. The UCI page that contains a more detailed description is Wine Quality.

In [26]:
# sklearn wants valid Python identifiers, so we replace spaces with underscores.
whitewines = pd.read_csv("whitewines.csv", sep=";")
whitewines.columns = whitewines.columns.str.replace(" ", "_")

print("First 5 rows:")
display(whitewines.head())
print(f"\nDimensions (rows, columns): {whitewines.shape}")
First 5 rows:
fixed_acidity volatile_acidity citric_acid residual_sugar chlorides free_sulfur_dioxide total_sulfur_dioxide density pH sulphates alcohol quality
0 7.0 0.27 0.36 20.7 0.045 45.0 170.0 1.0010 3.00 0.45 8.8 6
1 6.3 0.30 0.34 1.6 0.049 14.0 132.0 0.9940 3.30 0.49 9.5 6
2 8.1 0.28 0.40 6.9 0.050 30.0 97.0 0.9951 3.26 0.44 10.1 6
3 7.2 0.23 0.32 8.5 0.058 47.0 186.0 0.9956 3.19 0.40 9.9 6
4 7.2 0.23 0.32 8.5 0.058 47.0 186.0 0.9956 3.19 0.40 9.9 6
Dimensions (rows, columns): (4898, 12)

A Simple Regression Tree¶

In R the two-predictor illustration uses the tree package (tree(), plot(), partition.tree()). sklearn's DecisionTreeRegressor is the analogue of both tree and rpart; there is no separate partition.tree.

The syntax is not a formula. We pass a design matrix X and a response vector y.

In [27]:
# R: tree(quality ~ alcohol + volatile.acidity, data=whitewines)
feature_names = ["alcohol", "volatile_acidity"]
X_simple = whitewines[feature_names]
y = whitewines["quality"]

# sklearn grows until leaves are pure unless we stop it.
# tree() in R uses different defaults, so these two knobs are only
# to keep the illustration small and readable, not to match rpart's CP path.
simple_tree = DecisionTreeRegressor(
    max_depth=3,
    min_impurity_decrease=0.008,
    random_state=RANDOM_STATE,
)
simple_tree.fit(X_simple, y)

plt.figure(figsize=(10, 6))
plot_tree(
    simple_tree,
    feature_names=list(X_simple.columns),
    filled=True,
    rounded=True,
    fontsize=10,
)
plt.title("Simple regression tree for wine quality")
plt.show()
No description has been provided for this image

Rectangular partition¶

R draws the partition with partition.tree(). sklearn has no helper, so the next cell is a small translation of the tree package geometry: walk the fitted tree and draw the cut segments in the (alcohol, volatile_acidity) plane.

Points are colored by quality, as in the R figure (not by the vertical coordinate).

In [28]:
from matplotlib.collections import LineCollection


def _partition_segments(regressor, bbox, node=0):
    """Walk a two-predictor sklearn tree using child indices, not R's DFS vectors.

    bbox = [x_min, y_min, x_max, y_max]. Left child is {feature <= threshold}.
    """
    tree = regressor.tree_
    if tree.children_left[node] == -1:  # leaf
        return []

    feat = tree.feature[node]
    cut = tree.threshold[node]
    left = tree.children_left[node]
    right = tree.children_right[node]
    x_min, y_min, x_max, y_max = bbox
    segs = []

    if feat == 0:  # alcohol (x-axis)
        segs.append([(cut, y_min), (cut, y_max)])
        segs += _partition_segments(regressor, [x_min, y_min, cut, y_max], left)
        segs += _partition_segments(regressor, [cut, y_min, x_max, y_max], right)
    elif feat == 1:  # volatile_acidity (y-axis)
        segs.append([(x_min, cut), (x_max, cut)])
        segs += _partition_segments(regressor, [x_min, y_min, x_max, cut], left)
        segs += _partition_segments(regressor, [x_min, cut, x_max, y_max], right)
    else:
        raise ValueError("split on a variable other than the two plotted predictors")
    return segs


def partition_tree(regressor, data, y, ax=None, **scatter_kw):
    """Scatter two predictors and overlay the tree partition (R: partition.tree)."""
    x_min, y_min = data.min().to_numpy()
    x_max, y_max = data.max().to_numpy()
    pad_x = 0.025 * (x_max - x_min)
    pad_y = 0.025 * (y_max - y_min)
    bbox = [x_min - pad_x, y_min - pad_y, x_max + pad_x, y_max + pad_y]
    segs = _partition_segments(regressor, bbox)

    if ax is None:
        _, ax = plt.subplots()

    ax.scatter(data.iloc[:, 0], data.iloc[:, 1], c=y, **scatter_kw)
    ax.set_xlabel(data.columns[0])
    ax.set_ylabel(data.columns[1])
    ax.add_collection(LineCollection(segs, colors="k"))
    ax.autoscale()
    return ax
In [29]:
fig, ax = plt.subplots(figsize=(10, 6))
partition_tree(simple_tree, X_simple, y, ax, cmap="Greys", alpha=0.6, s=18)
ax.set_title("Partition in the (alcohol, volatile_acidity) plane")
plt.show()
No description has been provided for this image

R also calls prune.tree(..., best=4) to force four leaves. The sklearn analogue is to pick a ccp_alpha (next section) or to set max_leaf_nodes=4. We skip that extra plot; the pruned full-predictor tree below is the one that corresponds to prune() in rpart.

   

Fitting and Pruning a Regression Tree {-}¶

Unlike rpart, sklearn does not run cross-validation or prune while fitting. DecisionTreeRegressor() with default arguments grows a large tree $T_{\max}$ (the cp = 0 fit wine.tree2 in the notes). Pruning is a second step, using ccp_alpha.

sklearn's ccp_alpha is $\alpha$ itself in $R_\alpha(T)=R(T)+\alpha|T|$. The path of trees is the same idea but the printed numbers are not on the same scale.

In [30]:
X = whitewines.drop("quality", axis=1)
y = whitewines["quality"]

# Default sklearn tree = large T_max (R: control=list(cp=0)).
wine_tree_max = DecisionTreeRegressor(random_state=RANDOM_STATE)
wine_tree_max.fit(X, y)

# Full plot of T_max is unreadable; show the top layers only.
plt.figure(figsize=(15, 8))
plot_tree(
    wine_tree_max,
    max_depth=3,
    feature_names=list(X.columns),
    filled=True,
    rounded=True,
    fontsize=8,
)
plt.title("Large tree $T_{max}$ (top 3 levels)")
plt.show()
No description has been provided for this image

Reading the Tree¶

Start at the root and read down. The root is the mean of quality over the whole sample. The first split is on alcohol. The left child is the lower-alcohol rows (lower mean); the right child is the rest. Leaves sit at the bottom; shading tracks the predicted mean. Exact means will match this fitted sklearn tree. The greedy search is the same idea, the tie-breaking and default stopping rules are not.

Cross-validation and the $\alpha$ path¶

rpart builds the cost-complexity path and the CV table inside rpart(). In sklearn we ask for the path first, then run $K$-fold CV ourselves.

cost_complexity_pruning_path returns the increasing sequence of effective $\alpha$ values (the $\alpha_i$ from weakest-link pruning) and the corresponding total leaf impurity.

In [31]:
path = wine_tree_max.cost_complexity_pruning_path(X, y)
# Full path can have hundreds of alphas (here often ~900).
# 10-fold CV on every value is too slow for a lecture notebook.
# Keep ~40 alphas spaced along the path, including the endpoints.
ccp_alphas_full, impurities_full = path.ccp_alphas, path.impurities
idx = np.unique(np.linspace(0, len(ccp_alphas_full) - 1, 40).astype(int))
ccp_alphas = ccp_alphas_full[idx]
impurities = impurities_full[idx]

plt.figure(figsize=(8, 5))
plt.plot(ccp_alphas, impurities, marker="o", drawstyle="steps-post")
plt.xlabel(r"effective $\alpha$")
plt.ylabel("Total impurity of leaves")
plt.title(r"Training impurity vs. effective $\alpha$")
plt.show()
No description has been provided for this image

Choosing $\alpha$: minimum CV error and the 1-SE rule¶

As in the notes, use the minimum cross-validation error or the one-standard-error (1-SE) rule. We run 10-fold CV over the $\alpha$ path (xval = 10 in rpart.control).

GridSearchCV reports mean negative MSE, so larger scores are better. The 1-SE tree is the simplest tree (largest $\alpha$) whose CV score is within one standard error of the best score.

In [32]:
cv_fold = 10
tree_cv = GridSearchCV(
    DecisionTreeRegressor(random_state=RANDOM_STATE),
    param_grid={"ccp_alpha": ccp_alphas},
    cv=cv_fold,
    scoring="neg_mean_squared_error",
    n_jobs=-1,
)
tree_cv.fit(X, y)
Out[32]:
GridSearchCV(cv=10, estimator=DecisionTreeRegressor(random_state=598),
             n_jobs=-1,
             param_grid={'ccp_alpha': array([0.00000000e+00, 9.42299840e-05, 9.81562333e-05, 1.02082483e-04,
       1.36109977e-04, 1.36109977e-04, 1.36109977e-04, 1.53123724e-04,
       1.53123724e-04, 1.63331972e-04, 1.67519972e-04, 1.72525365e-04,
       1.78644345e-04, 1.81479969e-04, 1.85604514e-04, 1.93917303e-04,
       2.04164965e-04, 2.15654769e-04, 2.33331389e-04, 2.44997958e-04,
       2.52952117e-04, 2.72219954e-04, 2.83562452e-04, 2.99041626e-04,
       3.11108519e-04, 3.35039943e-04, 3.53362440e-04, 3.75654051e-04,
       3.97070833e-04, 4.29721220e-04, 4.77469705e-04, 5.35933034e-04,
       6.06219171e-04, 6.76528453e-04, 7.65618620e-04, 8.86789095e-04,
       1.06515657e-03, 1.36815182e-03, 1.99701311e-03, 1.26260584e-01])},
             scoring='neg_mean_squared_error')
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
estimator estimator: estimator object

This is assumed to implement the scikit-learn estimator interface.
Either estimator needs to provide a ``score`` function,
or ``scoring`` must be passed.
DecisionTreeR...dom_state=598)
param_grid param_grid: dict or list of dictionaries

Dictionary with parameters names (`str`) as keys and lists of
parameter settings to try as values, or a list of such
dictionaries, in which case the grids spanned by each dictionary
in the list are explored. This enables searching over any sequence
of parameter settings.
{'ccp_alpha': array([0.0000...26260584e-01])}
scoring scoring: str, callable, list, tuple or dict, default=None

Strategy to evaluate the performance of the cross-validated model on
the test set.

If `scoring` represents a single score, one can use:

- a single string (see :ref:`scoring_string_names`);
- a callable (see :ref:`scoring_callable`) that returns a single value;
- `None`, the `estimator`'s
:ref:`default evaluation criterion <scoring_api_overview>` is used.

If `scoring` represents multiple scores, one can use:

- a list or tuple of unique strings;
- a callable returning a dictionary where the keys are the metric
names and the values are the metric scores;
- a dictionary with metric names as keys and callables as values.

See :ref:`multimetric_grid_search` for an example.
'neg_mean_squared_error'
n_jobs n_jobs: int, default=None

Number of jobs to run in parallel.
``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.
``-1`` means using all processors. See :term:`Glossary <n_jobs>`
for more details.

.. versionchanged:: v0.20
`n_jobs` default changed from 1 to None
-1
cv cv: int, cross-validation generator or an iterable, default=None

Determines the cross-validation splitting strategy.
Possible inputs for cv are:

- None, to use the default 5-fold cross validation,
- integer, to specify the number of folds in a `(Stratified)KFold`,
- :term:`CV splitter`,
- an iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and ``y`` is
either binary or multiclass, :class:`StratifiedKFold` is used. In all
other cases, :class:`KFold` is used. These splitters are instantiated
with `shuffle=False` so the splits will be the same across calls.

Refer :ref:`User Guide <cross_validation>` for the various
cross-validation strategies that can be used here.

.. versionchanged:: 0.22
``cv`` default value if None changed from 3-fold to 5-fold.
10
refit refit: bool, str, or callable, default=True

Refit an estimator using the best found parameters on the whole
dataset.

For multiple metric evaluation, this needs to be a `str` denoting the
scorer that would be used to find the best parameters for refitting
the estimator at the end.

Where there are considerations other than maximum score in
choosing a best estimator, ``refit`` can be set to a function which
returns the selected ``best_index_`` given ``cv_results_``. In that
case, the ``best_estimator_`` and ``best_params_`` will be set
according to the returned ``best_index_`` while the ``best_score_``
attribute will not be available.

The refitted estimator is made available at the ``best_estimator_``
attribute and permits using ``predict`` directly on this
``GridSearchCV`` instance.

Also for multiple metric evaluation, the attributes ``best_index_``,
``best_score_`` and ``best_params_`` will only be available if
``refit`` is set and all of them will be determined w.r.t this specific
scorer.

See ``scoring`` parameter to know more about multiple metric
evaluation.

See :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_digits.py`
to see how to design a custom selection strategy using a callable
via `refit`.

See :ref:`this example
<sphx_glr_auto_examples_model_selection_plot_grid_search_refit_callable.py>`
for an example of how to use ``refit=callable`` to balance model
complexity and cross-validated score.

.. versionchanged:: 0.20
Support for callable added.
True
verbose verbose: int, default=0

Controls the verbosity of information printed during fitting, with higher
values yielding more detailed logging.

- 0 : no messages are printed;
- >=1 : summary of the total number of fits;
- >=2 : computation time for each fold and parameter candidate;
- >=3 : fold indices and scores;
- >=10 : parameter candidate indices and START messages before each fit.
0
pre_dispatch pre_dispatch: int, or str, default='2*n_jobs'

Controls the number of jobs that get dispatched during parallel
execution. Reducing this number can be useful to avoid an
explosion of memory consumption when more jobs get dispatched
than CPUs can process. This parameter can be:

- None, in which case all the jobs are immediately created and spawned. Use
this for lightweight and fast-running jobs, to avoid delays due to on-demand
spawning of the jobs
- An int, giving the exact number of total jobs that are spawned
- A str, giving an expression as a function of n_jobs, as in '2*n_jobs'
'2*n_jobs'
error_score error_score: 'raise' or numeric, default=np.nan

Value to assign to the score if an error occurs in estimator fitting.
If set to 'raise', the error is raised. If a numeric value is given,
FitFailedWarning is raised. This parameter does not affect the refit
step, which will always raise the error.
nan
return_train_score return_train_score: bool, default=False

If ``False``, the ``cv_results_`` attribute will not include training
scores.
Computing training scores is used to get insights on how different
parameter settings impact the overfitting/underfitting trade-off.
However computing the scores on the training set can be computationally
expensive and is not strictly required to select the parameters that
yield the best generalization performance.

.. versionadded:: 0.19

.. versionchanged:: 0.21
Default value was changed from ``True`` to ``False``
False
Fitted attributes
Name Type Value
best_estimator_ best_estimator_: estimator

Estimator that was chosen by the search, i.e. estimator
which gave highest score (or smallest loss if specified)
on the left out data. Not available if ``refit=False``.

See ``refit`` parameter for more information on allowed values.
DecisionTreeRegressor DecisionTreeR...dom_state=598)
best_index_ best_index_: int

The index (of the ``cv_results_`` arrays) which corresponds to the best
candidate parameter setting.

The dict at ``search.cv_results_['params'][search.best_index_]`` gives
the parameter setting for the best model, that gives the highest
mean score (``search.best_score_``).

For multi-metric evaluation, this is present only if ``refit`` is
specified.
int64 np.int64(38)
best_params_ best_params_: dict

Parameter setting that gave the best results on the hold out data.

For multi-metric evaluation, this is present only if ``refit`` is
specified.
dict {'cc...ha': np.float64(0....7013105639954)}
best_score_ best_score_: float

Mean cross-validated score of the best_estimator

For multi-metric evaluation, this is present only if ``refit`` is
specified.

This attribute is not available if ``refit`` is a function.
float64 -0.5892
cv_results_ cv_results_: dict of numpy (masked) ndarrays

A dict with keys as column headers and values as columns, that can be
imported into a pandas ``DataFrame``.

For instance the below given table

+------------+-----------+------------+-----------------+---+---------+
|param_kernel|param_gamma|param_degree|split0_test_score|...|rank_t...|
+============+===========+============+=================+===+=========+
| 'poly' | -- | 2 | 0.80 |...| 2 |
+------------+-----------+------------+-----------------+---+---------+
| 'poly' | -- | 3 | 0.70 |...| 4 |
+------------+-----------+------------+-----------------+---+---------+
| 'rbf' | 0.1 | -- | 0.80 |...| 3 |
+------------+-----------+------------+-----------------+---+---------+
| 'rbf' | 0.2 | -- | 0.93 |...| 1 |
+------------+-----------+------------+-----------------+---+---------+

will be represented by a ``cv_results_`` dict of::

{
'param_kernel': masked_array(data = ['poly', 'poly', 'rbf', 'rbf'],
mask = [False False False False]...)
'param_gamma': masked_array(data = [-- -- 0.1 0.2],
mask = [ True True False False]...),
'param_degree': masked_array(data = [2.0 3.0 -- --],
mask = [False False True True]...),
'split0_test_score' : [0.80, 0.70, 0.80, 0.93],
'split1_test_score' : [0.82, 0.50, 0.70, 0.78],
'mean_test_score' : [0.81, 0.60, 0.75, 0.85],
'std_test_score' : [0.01, 0.10, 0.05, 0.08],
'rank_test_score' : [2, 4, 3, 1],
'split0_train_score' : [0.80, 0.92, 0.70, 0.93],
'split1_train_score' : [0.82, 0.55, 0.70, 0.87],
'mean_train_score' : [0.81, 0.74, 0.70, 0.90],
'std_train_score' : [0.01, 0.19, 0.00, 0.03],
'mean_fit_time' : [0.73, 0.63, 0.43, 0.49],
'std_fit_time' : [0.01, 0.02, 0.01, 0.01],
'mean_score_time' : [0.01, 0.06, 0.04, 0.04],
'std_score_time' : [0.00, 0.00, 0.00, 0.01],
'params' : [{'kernel': 'poly', 'degree': 2}, ...],
}

For an example of visualization and interpretation of GridSearch results,
see :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_stats.py`.

NOTE

The key ``'params'`` is used to store a list of parameter
settings dicts for all the parameter candidates.

The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and
``std_score_time`` are all in seconds.

For multi-metric evaluation, the scores for all the scorers are
available in the ``cv_results_`` dict at the keys ending with that
scorer's name (``'_<scorer_name>'``) instead of ``'_score'`` shown
above. ('split0_test_precision', 'mean_train_precision' etc.)
dict {'me...me': array([0.0437..., 0.02228172]), 'me...me': array([0.0012..., 0.00091493]), 'me...re': array([-1.021... -0.75812172]), 'pa...ha': masked_array(...l_value=1e+20), ...}
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,)

Names of features seen during :term:`fit`. Only defined if
`best_estimator_` is defined (see the documentation for the `refit`
parameter for more details) and that `best_estimator_` exposes
`feature_names_in_` when fit.

.. versionadded:: 1.0
ndarray[object](11,) ['fixed_acidity','volatile_acidity','citric_acid',...,'pH','sulphates', 'alcohol']
multimetric_ multimetric_: bool

Whether or not the scorers compute several metrics.
bool False
n_features_in_ n_features_in_: int

Number of features seen during :term:`fit`. Only defined if
`best_estimator_` is defined (see the documentation for the `refit`
parameter for more details) and that `best_estimator_` exposes
`n_features_in_` when fit.

.. versionadded:: 0.24
int 11
n_splits_ n_splits_: int

The number of cross-validation splits (folds/iterations).
int 10
refit_time_ refit_time_: float

Seconds used for refitting the best model on the whole dataset.

This is present only if ``refit`` is not False.

.. versionadded:: 0.20
float 0.0208
scorer_ scorer_: function or a dict

Scorer function used on the held out data to choose the best
parameters for the model.

For multi-metric evaluation, this attribute holds the validated
``scoring`` dict which maps the scorer key to the scorer callable.
_Scorer make_scorer(m...hod='predict')
DecisionTreeRegressor(ccp_alpha=np.float64(0.001997013105639954),
                      random_state=598)
Parameters
random_state random_state: int, RandomState instance or None, default=None

Controls the randomness of the estimator. The features are always
randomly permuted at each split, even if ``splitter`` is set to
``"best"``. When ``max_features < n_features``, the algorithm will
select ``max_features`` at random at each split before finding the best
split among them. But the best found split may vary across different
runs, even if ``max_features=n_features``. That is the case, if the
improvement of the criterion is identical for several splits and one
split has to be selected at random. To obtain a deterministic behaviour
during fitting, ``random_state`` has to be fixed to an integer.
See :term:`Glossary <random_state>` for details.
598
ccp_alpha ccp_alpha: non-negative float, default=0.0

Complexity parameter used for Minimal Cost-Complexity Pruning. The
subtree with the largest cost complexity that is smaller than
``ccp_alpha`` will be chosen. By default, no pruning is performed. See
:ref:`minimal_cost_complexity_pruning` for details. See
:ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py`
for an example of such pruning.

.. versionadded:: 0.22
np.float64(0....7013105639954)
criterion criterion: {"squared_error", "absolute_error", "poisson"}, default="squared_error"

The function to measure the quality of a split. Supported criteria
are "squared_error" for the mean squared error, which is equal to
variance reduction as feature selection criterion and minimizes the L2
loss using the mean of each terminal node, "absolute_error" for the mean
absolute error, which minimizes the L1 loss using the median of each terminal
node, and "poisson" which uses reduction in Poisson deviance to find splits,
also using the mean of each terminal node.

.. versionadded:: 0.18
Mean Absolute Error (MAE) criterion.

.. versionadded:: 0.24
Poisson deviance criterion.

.. versionchanged:: 1.9
Criterion `"friedman_mse"` was deprecated.
'squared_error'
splitter splitter: {"best", "random"}, default="best"

The strategy used to choose the split at each node. Supported
strategies are "best" to choose the best split and "random" to choose
the best random split.
'best'
max_depth max_depth: int, default=None

The maximum depth of the tree. If None, then nodes are expanded until
all leaves are pure or until all leaves contain less than
min_samples_split samples.

For an example of how ``max_depth`` influences the model, see
:ref:`sphx_glr_auto_examples_tree_plot_tree_regression.py`.
None
min_samples_split min_samples_split: int or float, default=2

The minimum number of samples required to split an internal node:

- If int, then consider `min_samples_split` as the minimum number.
- If float, then `min_samples_split` is a fraction and
`ceil(min_samples_split * n_samples)` are the minimum
number of samples for each split.

.. versionchanged:: 0.18
Added float values for fractions.
2
min_samples_leaf min_samples_leaf: int or float, default=1

The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at
least ``min_samples_leaf`` training samples in each of the left and
right branches. This may have the effect of smoothing the model,
especially in regression.

- If int, then consider `min_samples_leaf` as the minimum number.
- If float, then `min_samples_leaf` is a fraction and
`ceil(min_samples_leaf * n_samples)` are the minimum
number of samples for each node.

.. versionchanged:: 0.18
Added float values for fractions.
1
min_weight_fraction_leaf min_weight_fraction_leaf: float, default=0.0

The minimum weighted fraction of the sum total of weights (of all
the input samples) required to be at a leaf node. Samples have
equal weight when sample_weight is not provided.
0.0
max_features max_features: int, float or {"sqrt", "log2"}, default=None

The number of features to consider when looking for the best split:

- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
`max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
- If None, then `max_features=n_features`.

Note: the search for a split does not stop until at least one
valid partition of the node samples is found, even if it requires to
effectively inspect more than ``max_features`` features.
None
max_leaf_nodes max_leaf_nodes: int, default=None

Grow a tree with ``max_leaf_nodes`` in best-first fashion.
Best nodes are defined as relative reduction in impurity.
If None then unlimited number of leaf nodes.
None
min_impurity_decrease min_impurity_decrease: float, default=0.0

A node will be split if this split induces a decrease of the impurity
greater than or equal to this value.

The weighted impurity decrease equation is the following::

N_t / N * (impurity - N_t_R / N_t * right_impurity
- N_t_L / N_t * left_impurity)

where ``N`` is the total number of samples, ``N_t`` is the number of
samples at the current node, ``N_t_L`` is the number of samples in the
left child, and ``N_t_R`` is the number of samples in the right child.

``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
if ``sample_weight`` is passed.

.. versionadded:: 0.19
0.0
monotonic_cst monotonic_cst: array-like of int of shape (n_features), default=None

Indicates the monotonicity constraint to enforce on each feature.
- 1: monotonic increase
- 0: no constraint
- -1: monotonic decrease

If monotonic_cst is None, no constraints are applied.

Monotonicity constraints are not supported for:
- multioutput regressions (i.e. when `n_outputs_ > 1`).

Read more in the :ref:`User Guide <monotonic_cst_gbdt>`.

.. versionadded:: 1.4
None
Fitted attributes
Name Type Value
feature_importances_ feature_importances_: ndarray of shape (n_features,)

The feature importances.
The higher, the more important the feature.
The importance of a feature is computed as the
(normalized) total reduction of the criterion brought
by that feature. It is also known as the Gini importance [4]_.

Warning: impurity-based feature importances can be misleading for
high cardinality features (many unique values). See
:func:`sklearn.inspection.permutation_importance` as an alternative.
ndarray[float64](11,) [0.02,0.19,0.03,...,0.03,0.01,0.52]
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,)

Names of features seen during :term:`fit`. Defined only when `X`
has feature names that are all strings.

.. versionadded:: 1.0
ndarray[object](11,) ['fixed_acidity','volatile_acidity','citric_acid',...,'pH','sulphates', 'alcohol']
max_features_ max_features_: int

The inferred value of max_features.
int 11
n_features_in_ n_features_in_: int

Number of features seen during :term:`fit`.

.. versionadded:: 0.24
int 11
n_outputs_ n_outputs_: int

The number of outputs when ``fit`` is performed.
int 1
tree_ tree_: Tree instance

The underlying Tree object. Please refer to
``help(sklearn.tree._tree.Tree)`` for attributes of Tree object and
:ref:`sphx_glr_auto_examples_tree_plot_unveil_tree_structure.py`
for basic usage of these attributes.
Tree <sklearn.tree...t 0x11a6c98b0>
In [33]:
# Relative CV error, analogous to rpart's xerror
# (divide by the CV error of the largest-alpha tree = the root).
cv_mse = -tree_cv.cv_results_["mean_test_score"]
cv_mse_rel = cv_mse / cv_mse[-1]

plt.figure(figsize=(8, 5))
plt.plot(ccp_alphas, cv_mse_rel, marker="o", drawstyle="steps-post")
plt.xlabel(r"$\alpha$")
plt.ylabel("Relative CV MSE")
plt.title("Cost-complexity path (Python stand-in for plotcp)")
plt.show()
No description has been provided for this image
In [34]:
best_alpha = tree_cv.best_params_["ccp_alpha"]
best_idx = tree_cv.best_index_
best_score = tree_cv.best_score_  # negative MSE
print(f"alpha at minimum CV error: {best_alpha:.6f}")

# SE of the mean CV score. GridSearchCV's std_test_score is the
# standard deviation across folds; divide by sqrt(K) for the SE.
cv_scores = tree_cv.cv_results_["mean_test_score"]
best_se = tree_cv.cv_results_["std_test_score"][best_idx] / np.sqrt(cv_fold)
threshold = best_score - best_se

# ccp_alphas increase with simplicity. Take the last alpha still within 1 SE.
one_se_idx = np.where(cv_scores >= threshold)[0][-1]
one_se_alpha = ccp_alphas[one_se_idx]
print(f"1-SE alpha: {one_se_alpha:.6f}")
alpha at minimum CV error: 0.001997
1-SE alpha: 0.001997
In [35]:
wine_tree_pruned = DecisionTreeRegressor(
    random_state=RANDOM_STATE, ccp_alpha=one_se_alpha
)
wine_tree_pruned.fit(X, y)

plt.figure(figsize=(10, 6))
plot_tree(
    wine_tree_pruned,
    feature_names=list(X.columns),
    filled=True,
    rounded=True,
    fontsize=8,
)
plt.title(rf"Pruned tree (1-SE $\alpha$ = {one_se_alpha:.4f})")
plt.show()
No description has been provided for this image

After the tree is fit, predict() produces fitted values on new or existing rows.

  

Random Forests in Python¶

Load the forest estimator and hold out a test set. R used sample() with seed 598 and a 30% holdout. train_test_split(..., random_state=598, test_size=0.3) is the same fraction and the same seed integer, but it is not the same partition as R's sample(), so the MSEs will not match the knitted R output exactly.

In [36]:
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=RANDOM_STATE
)
print(f"Train rows: {X_train.shape[0]}, test rows: {X_test.shape[0]}")
Train rows: 3428, test rows: 1470

Model fitting¶

RandomForestRegressor is the analogue of randomForest(). Note that oob_score_ itself is an OOB $R^2$, not an MSE. For a number comparable to the R notes we compute MSE from oob_prediction_.

In [37]:
rf_wines = RandomForestRegressor(
    n_estimators=400,
    max_features=1 / 3,   # p/3, the usual regression default
    oob_score=True,
    random_state=RANDOM_STATE,
    n_jobs=-1,
)
rf_wines.fit(X_train, y_train)
print(f"max_features used: {rf_wines.max_features}")
print(f"mtry in the R notes was 3; here p/3 = {X_train.shape[1] / 3:.2f}")
max_features used: 0.3333333333333333
mtry in the R notes was 3; here p/3 = 3.67

Test-set mean squared error:

In [38]:
test_mse = mean_squared_error(y_test, rf_wines.predict(X_test))
print(f"Test MSE: {test_mse:.4f}")
Test MSE: 0.3912

There are two training-side errors: the error of predict() on the training rows, and the out-of-bag error stored on the forest.

In [39]:
train_mse = mean_squared_error(y_train, rf_wines.predict(X_train))
oob_mse = mean_squared_error(y_train, rf_wines.oob_prediction_)
print(f"Training MSE (in-sample predict): {train_mse:.4f}")
print(f"OOB MSE:                          {oob_mse:.4f}")
Training MSE (in-sample predict): 0.0498
OOB MSE:                          0.3700

The in-sample training error is much smaller than the test error. The OOB error is closer to the test error and is a more reliable estimate of generalization performance.

Performance vs. Number of Trees¶

R's plot(rf) is OOB MSE against the number of trees. sklearn does not produce that plot. The next cell rebuilds a running OOB average from the individual trees. It uses _generate_unsampled_indices, an internal helper that mirrors how the forest drew each bootstrap sample. Treat it as a plotting device, not as public API.

In [40]:
from sklearn.ensemble._forest import _generate_unsampled_indices

n_samples = X_train.shape[0]
n_trees = rf_wines.n_estimators
oob_preds = np.full((n_samples, n_trees), np.nan)

for i, tree in enumerate(rf_wines.estimators_):
    oob_idx = _generate_unsampled_indices(tree.random_state, n_samples, n_samples, None)
    oob_preds[oob_idx, i] = tree.predict(X_train.iloc[oob_idx].to_numpy())

oob_count = np.cumsum(~np.isnan(oob_preds), axis=1)
with np.errstate(divide="ignore", invalid="ignore"):
    oob_mean = np.nancumsum(oob_preds, axis=1) / oob_count

oob_mse_path = np.nanmean((oob_mean - y_train.to_numpy()[:, None]) ** 2, axis=0)

plt.figure(figsize=(8, 5))
plt.plot(np.arange(1, n_trees + 1), oob_mse_path)
plt.xlabel("Number of trees")
plt.ylabel("OOB MSE")
plt.title("Random forest OOB error vs. number of trees")
plt.show()
No description has been provided for this image

The error usually drops quickly and then levels off. Extra trees rarely make the forest worse; they may simply be unnecessary.

Variable Importance¶

R reports both %IncMSE (permutation) and IncNodePurity (RSS / impurity gain). rf_wines.feature_importances_ is the impurity measure only. Permutation importance is computed separately with permutation_importance on the OOB-style held-out rows -- here we use the test set for a clean comparison.

In [41]:
impurity = pd.Series(rf_wines.feature_importances_, index=X.columns).sort_values()

perm = permutation_importance(
    rf_wines, X_test, y_test,
    n_repeats=10,
    random_state=RANDOM_STATE,
    n_jobs=-1,
)
perm_imp = pd.Series(perm.importances_mean, index=X.columns).sort_values()

fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].barh(perm_imp.index, perm_imp.values)
axes[0].set_title("Permutation importance")
axes[0].set_xlabel("Increase in test MSE")
axes[1].barh(impurity.index, impurity.values)
axes[1].set_title("Impurity importance")
axes[1].set_xlabel("Mean decrease in impurity")
fig.tight_layout()
plt.show()
No description has been provided for this image

Gradient Boosting Machines in Python¶

A GBM is a sum of trees built sequentially. For squared error, tree $t$ is fit to the current residuals and added with a learning rate $\lambda$:

$$ F_t(x) = F_{t-1}(x) + \lambda\,\hat h_t(x). $$

The gbm arguments and their sklearn names:

sklearn (GradientBoostingRegressor) Role
loss="squared_error" (default) squared error
n_estimators number of trees $T$
learning_rate $\lambda$ (default 0.1 in both)
max_depth R default is 1 (stumps); sklearn default is 3
subsample R default 0.5; sklearn default 1.0 (no subsampling)
none gbm.perf uses CV; sklearn has no built-in CV curve. We plot test MSE vs. $T$ with staged_predict.

The first specification below uses no shrinkage and no subsampling, so it is close to unregularized stagewise fitting. That is a poor default; we use it only as a baseline, matching the R notes.

In [42]:
gbm_wines = GradientBoostingRegressor(
    n_estimators=100,
    learning_rate=1.0,
    max_depth=3,
    subsample=1.0,
    random_state=RANDOM_STATE,
)
gbm_wines.fit(X_train, y_train)
Out[42]:
GradientBoostingRegressor(learning_rate=1.0, random_state=598)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Parameters
learning_rate learning_rate: float, default=0.1

Learning rate shrinks the contribution of each tree by `learning_rate`.
There is a trade-off between learning_rate and n_estimators.
Values must be in the range `[0.0, inf)`.
1.0
random_state random_state: int, RandomState instance or None, default=None

Controls the random seed given to each Tree estimator at each
boosting iteration.
In addition, it controls the random permutation of the features at
each split (see Notes for more details).
It also controls the random splitting of the training data to obtain a
validation set if `n_iter_no_change` is not None.
Pass an int for reproducible output across multiple function calls.
See :term:`Glossary <random_state>`.
598
loss loss: {'squared_error', 'absolute_error', 'huber', 'quantile'}, default='squared_error'

Loss function to be optimized. 'squared_error' refers to the squared
error for regression. 'absolute_error' refers to the absolute error of
regression and is a robust loss function. 'huber' is a
combination of the two. 'quantile' allows quantile regression (use
`alpha` to specify the quantile).
See
:ref:`sphx_glr_auto_examples_ensemble_plot_gradient_boosting_quantile.py`
for an example that demonstrates quantile regression for creating
prediction intervals with `loss='quantile'`.
'squared_error'
n_estimators n_estimators: int, default=100

The number of boosting stages to perform. Gradient boosting
is fairly robust to over-fitting so a large number usually
results in better performance.
Values must be in the range `[1, inf)`.
100
subsample subsample: float, default=1.0

The fraction of samples to be used for fitting the individual base
learners. If smaller than 1.0 this results in Stochastic Gradient
Boosting. `subsample` interacts with the parameter `n_estimators`.
Choosing `subsample < 1.0` leads to a reduction of variance
and an increase in bias.
Values must be in the range `(0.0, 1.0]`.
1.0
criterion criterion: {'friedman_mse', 'squared_error'}, default='friedman_mse'

This parameter has no effect.

.. versionadded:: 0.18

.. deprecated:: 1.9
`criterion` is deprecated and will be removed in 1.11.
'deprecated'
min_samples_split min_samples_split: int or float, default=2

The minimum number of samples required to split an internal node:

- If int, values must be in the range `[2, inf)`.
- If float, values must be in the range `(0.0, 1.0]` and `min_samples_split`
will be `ceil(min_samples_split * n_samples)`.

.. versionchanged:: 0.18
Added float values for fractions.
2
min_samples_leaf min_samples_leaf: int or float, default=1

The minimum number of samples required to be at a leaf node.
A split point at any depth will only be considered if it leaves at
least ``min_samples_leaf`` training samples in each of the left and
right branches. This may have the effect of smoothing the model,
especially in regression.

- If int, values must be in the range `[1, inf)`.
- If float, values must be in the range `(0.0, 1.0)` and `min_samples_leaf`
will be `ceil(min_samples_leaf * n_samples)`.

.. versionchanged:: 0.18
Added float values for fractions.
1
min_weight_fraction_leaf min_weight_fraction_leaf: float, default=0.0

The minimum weighted fraction of the sum total of weights (of all
the input samples) required to be at a leaf node. Samples have
equal weight when sample_weight is not provided.
Values must be in the range `[0.0, 0.5]`.
0.0
max_depth max_depth: int or None, default=3

Maximum depth of the individual regression estimators. The maximum
depth limits the number of nodes in the tree. Tune this parameter
for best performance; the best value depends on the interaction
of the input variables. If None, then nodes are expanded until
all leaves are pure or until all leaves contain less than
min_samples_split samples.
If int, values must be in the range `[1, inf)`.
3
min_impurity_decrease min_impurity_decrease: float, default=0.0

A node will be split if this split induces a decrease of the impurity
greater than or equal to this value.
Values must be in the range `[0.0, inf)`.

The weighted impurity decrease equation is the following::

N_t / N * (impurity - N_t_R / N_t * right_impurity
- N_t_L / N_t * left_impurity)

where ``N`` is the total number of samples, ``N_t`` is the number of
samples at the current node, ``N_t_L`` is the number of samples in the
left child, and ``N_t_R`` is the number of samples in the right child.

``N``, ``N_t``, ``N_t_R`` and ``N_t_L`` all refer to the weighted sum,
if ``sample_weight`` is passed.

.. versionadded:: 0.19
0.0
init init: estimator or 'zero', default=None

An estimator object that is used to compute the initial predictions.
``init`` has to provide :term:`fit` and :term:`predict`. If 'zero', the
initial raw predictions are set to zero. By default a
``DummyEstimator`` is used, predicting either the average target value
(for loss='squared_error'), or a quantile for the other losses.
None
max_features max_features: {'sqrt', 'log2'}, int or float, default=None

The number of features to consider when looking for the best split:

- If int, values must be in the range `[1, inf)`.
- If float, values must be in the range `(0.0, 1.0]` and the features
considered at each split will be `max(1, int(max_features * n_features_in_))`.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
- If None, then `max_features=n_features`.

Choosing `max_features < n_features` leads to a reduction of variance
and an increase in bias.

Note: the search for a split does not stop until at least one
valid partition of the node samples is found, even if it requires to
effectively inspect more than ``max_features`` features.
None
alpha alpha: float, default=0.9

The alpha-quantile of the huber loss function and the quantile
loss function. Only if ``loss='huber'`` or ``loss='quantile'``.
Values must be in the range `(0.0, 1.0)`.
0.9
verbose verbose: int, default=0

Enable verbose output. If 1 then it prints progress and performance
once in a while (the more trees the lower the frequency). If greater
than 1 then it prints progress and performance for every tree.
Values must be in the range `[0, inf)`.
0
max_leaf_nodes max_leaf_nodes: int, default=None

Grow trees with ``max_leaf_nodes`` in best-first fashion.
Best nodes are defined as relative reduction in impurity.
Values must be in the range `[2, inf)`.
If None, then unlimited number of leaf nodes.
None
warm_start warm_start: bool, default=False

When set to ``True``, reuse the solution of the previous call to fit
and add more estimators to the ensemble, otherwise, just erase the
previous solution. See :term:`the Glossary <warm_start>`.
False
validation_fraction validation_fraction: float, default=0.1

The proportion of training data to set aside as validation set for
early stopping. Values must be in the range `(0.0, 1.0)`.
Only used if ``n_iter_no_change`` is set to an integer.

.. versionadded:: 0.20
0.1
n_iter_no_change n_iter_no_change: int, default=None

``n_iter_no_change`` is used to decide if early stopping will be used
to terminate training when validation score is not improving. By
default it is set to None to disable early stopping. If set to a
number, it will set aside ``validation_fraction`` size of the training
data as validation and terminate training when validation score is not
improving in all of the previous ``n_iter_no_change`` numbers of
iterations.
Values must be in the range `[1, inf)`.
See
:ref:`sphx_glr_auto_examples_ensemble_plot_gradient_boosting_early_stopping.py`.

.. versionadded:: 0.20
None
tol tol: float, default=1e-4

Tolerance for the early stopping. When the loss is not improving
by at least tol for ``n_iter_no_change`` iterations (if set to a
number), the training stops.
Values must be in the range `[0.0, inf)`.

.. versionadded:: 0.20
0.0001
ccp_alpha ccp_alpha: non-negative float, default=0.0

Complexity parameter used for Minimal Cost-Complexity Pruning. The
subtree with the largest cost complexity that is smaller than
``ccp_alpha`` will be chosen. By default, no pruning is performed.
Values must be in the range `[0.0, inf)`.
See :ref:`minimal_cost_complexity_pruning` for details. See
:ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py`
for an example of such pruning.

.. versionadded:: 0.22
0.0
Fitted attributes
Name Type Value
estimators_ estimators_: ndarray of DecisionTreeRegressor of shape (n_estimators, 1)

The collection of fitted sub-estimators.
ndarray[object](100, 1) [[DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)], [DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)], [DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)], ..., [DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)], [DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)], [DecisionTreeRegressor(max_depth=3, random_state=RandomState(MT19937) at 0x11A09CD40)]]
feature_importances_ feature_importances_: ndarray of shape (n_features,)

The impurity-based feature importances.
The higher, the more important the feature.
The importance of a feature is computed as the (normalized)
total reduction of the MSE brought by that feature. It is also
known as the Gini importance.

Warning: impurity-based feature importances can be misleading for
high cardinality features (many unique values). See
:func:`sklearn.inspection.permutation_importance` as an alternative.
ndarray[float64](11,) [0.04,0.15,0.04,...,0.06,0.04,0.3 ]
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,)

Names of features seen during :term:`fit`. Defined only when `X`
has feature names that are all strings.

.. versionadded:: 1.0
ndarray[object](11,) ['fixed_acidity','volatile_acidity','citric_acid',...,'pH','sulphates', 'alcohol']
init_ init_: estimator

The estimator that provides the initial predictions. Set via the ``init``
argument.
DummyRegressor DummyRegressor()
max_features_ max_features_: int

The inferred value of max_features.
int 11
n_estimators_ n_estimators_: int

The number of estimators as selected by early stopping (if
``n_iter_no_change`` is specified). Otherwise it is set to
``n_estimators``.
int 100
n_features_in_ n_features_in_: int

Number of features seen during :term:`fit`.

.. versionadded:: 0.24
int 11
n_trees_per_iteration_ n_trees_per_iteration_: int

The number of trees that are built at each iteration. For regressors, this is
always 1.

.. versionadded:: 1.4.0
int 1
train_score_ train_score_: ndarray of shape (n_estimators,)

The i-th score ``train_score_[i]`` is the loss of the
model at iteration ``i`` on the in-bag sample.
If ``subsample == 1`` this is the loss on the training data.
ndarray[float64](100,) [0.55,0.52,0.5 ,...,0.18,0.18,0.18]

Number of trees¶

gbm.perf picks $T$ from a CV curve. sklearn does not. train_score_ is the training loss after each tree; staged_predict gives the test path. With $\lambda=1$, the test-MSE minimum is often early.

In [43]:
train_err = gbm_wines.train_score_  # training MSE path
test_err = np.array([
    mean_squared_error(y_test, y_hat)
    for y_hat in gbm_wines.staged_predict(X_test)
])
opt_size = int(np.argmin(test_err) + 1)

plt.figure(figsize=(10, 6))
plt.plot(np.arange(1, len(train_err) + 1), train_err, label="Training MSE")
plt.plot(np.arange(1, len(test_err) + 1), test_err, label="Test MSE")
plt.axvline(opt_size, linestyle="--", label=f"Min test MSE at T = {opt_size}")
plt.xlabel("Number of trees")
plt.ylabel("MSE")
plt.title(r"GBM baseline ($\lambda = 1$, no subsampling)")
plt.legend()
plt.show()
No description has been provided for this image

A more standard specification: small learning rate, more trees, and subsample=0.5 (bag.fraction = 0.5).

In [44]:
gbm_wines2 = GradientBoostingRegressor(
    n_estimators=1000,
    learning_rate=0.1,
    max_depth=3,
    subsample=0.5,
    random_state=RANDOM_STATE,
)
gbm_wines2.fit(X_train, y_train)

train_err2 = gbm_wines2.train_score_
test_err2 = np.array([
    mean_squared_error(y_test, y_hat)
    for y_hat in gbm_wines2.staged_predict(X_test)
])
opt_size2 = int(np.argmin(test_err2) + 1)

plt.figure(figsize=(10, 6))
plt.plot(np.arange(1, len(train_err2) + 1), train_err2, label="Training MSE")
plt.plot(np.arange(1, len(test_err2) + 1), test_err2, label="Test MSE")
plt.axvline(opt_size2, linestyle="--", label=f"Min test MSE at T = {opt_size2}")
plt.xlabel("Number of trees")
plt.ylabel("MSE")
plt.title(r"GBM with $\lambda = 0.1$ and subsample $= 0.5$")
plt.legend()
plt.show()
No description has been provided for this image

Variable Importance¶

feature_importances_ on a GBM is the analogue of gbm's relative influence (normalized total improvement from splits on that variable). A permutation measure is available as in the forest section; we plot both on the second, more standard fit.

In [45]:
rel_inf = pd.Series(gbm_wines2.feature_importances_, index=X.columns).sort_values()

perm_gbm = permutation_importance(
    gbm_wines2, X_test, y_test,
    n_repeats=10,
    random_state=RANDOM_STATE,
    n_jobs=-1,
)
perm_gbm_s = pd.Series(perm_gbm.importances_mean, index=X.columns).sort_values()

fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].barh(rel_inf.index, rel_inf.values)
axes[0].set_title("Relative influence")
axes[0].set_xlabel("Importance")
axes[1].barh(perm_gbm_s.index, perm_gbm_s.values)
axes[1].set_title("Permutation importance")
axes[1].set_xlabel("Increase in test MSE")
fig.tight_layout()
plt.show()
No description has been provided for this image