9.5b SVM Examples in Python¶

This notebook is the Python version of the lab SVM Examples in R in the lecture notes. The code shows how the objects from the notes — $\beta$, $C$, support vectors, kernels — appear in a fitted SVM.

The calculations use sklearn.svm.SVC. Its argument C is the $C$ of the notes (the same quantity that e1071::svm() calls cost). We do not scale the toy examples, so that coefficients and plots stay in the original units of $X$. In applications one would usually scale the features.

Three comparisons:

  • a hard margin versus a soft margin (what $C$ does);

  • a linear SVM versus logistic regression on the same points (who influences the fit);

  • a linear kernel versus a radial kernel, with a Bayes boundary as a reference.

Linear separable case: maximal margin classifier¶

We start with a small maximal-margin example (adapted from an ISLR exercise).

In [12]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

plt.rcParams["figure.dpi"] = 120

toy = pd.DataFrame({
    "X1": [3, 2, 4, 1, 2, 4, 4],
    "X2": [4, 2, 4, 4, 1, 3, 1],
    "Y":  ["red", "red", "red", "red", "blue", "blue", "blue"],
})
toy
Out[12]:
X1 X2 Y
0 3 4 red
1 2 2 red
2 4 4 red
3 1 4 red
4 2 1 blue
5 4 3 blue
6 4 1 blue

A scatterplot of the data:

In [13]:
fig, ax = plt.subplots(figsize=(5, 5))
ax.scatter(toy["X1"], toy["X2"], c=toy["Y"], s=60)
ax.set_xlim(0.5, 4.5)
ax.set_ylim(0.5, 4.5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_aspect("equal")
plt.show()
No description has been provided for this image

The linear separating hyperplane has the form

$$ \beta_0 + \beta_1 X_1 + \beta_2 X_2 = 0. $$

We compute it from a hard-margin SVM fit by taking a very large C:

In [14]:
X_toy = toy[["X1", "X2"]].to_numpy()
y_toy = toy["Y"].to_numpy()

svm_toy = SVC(kernel="linear", C=1e5)
svm_toy.fit(X_toy, y_toy)

# sklearn stores beta as coef_ and beta0 as intercept_
beta = svm_toy.coef_.ravel()
beta0 = float(svm_toy.intercept_[0])
beta, beta0
Out[14]:
(array([-1.99941406,  1.99941406]), 0.9997070312500023)

So the fitted hyperplane is $\hat\beta_0 + \hat\beta_1 X_1 + \hat\beta_2 X_2 = 0$. The exact numbers can differ in sign from the R lab because sklearn codes the two classes in a fixed order; the line is the same.

The decision boundary in slope–intercept form and the margin width $2/\|\beta\|$:

In [15]:
slope = -beta[0] / beta[1]
intercept = -beta0 / beta[1]
margin = 2 / np.linalg.norm(beta)
slope, intercept, margin
Out[15]:
(np.float64(1.0000000000000004),
 np.float64(-0.5000000000000011),
 np.float64(0.7073140020855958))

A new point $x^*$ is assigned to a class according to $\operatorname{sign}(f(x^*))$. The solid line below is the separating hyperplane; the dotted lines are the margins.

The fit used a large C and no scaling, so this is, for practical purposes, the hard-margin solution in the original coordinates. In sklearn, support_ holds the (0-based) indices of the support vectors and n_support_ is the count by class. The intercept is intercept_, not the hard-margin midpoint formula from the notes.

In [16]:
xx = np.linspace(0.5, 4.5, 200)
fig, ax = plt.subplots(figsize=(5, 5))

xg, yg = np.meshgrid(xx, xx)
grid = np.column_stack([xg.ravel(), yg.ravel()])
Z = svm_toy.decision_function(grid).reshape(xg.shape)
ax.contourf(xg, yg, Z, levels=[-1e9, 0, 1e9], colors=["#F7E7E7", "#E6F2FA"], alpha=0.8)
ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=1.5)
ax.contour(xg, yg, Z, levels=[-1, 1], colors="gray", linestyles="dotted")

ax.scatter(toy["X1"], toy["X2"], c=toy["Y"], s=60, zorder=3)
ax.scatter(
    X_toy[svm_toy.support_, 0],
    X_toy[svm_toy.support_, 1],
    facecolors="none", edgecolors="black", s=140, linewidths=1.2, zorder=4,
)
ax.set_xlim(0.5, 4.5)
ax.set_ylim(0.5, 4.5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_aspect("equal")
plt.show()

print("support_ (0-based indices):", svm_toy.support_)
print("n_support_:", svm_toy.n_support_)
No description has been provided for this image
support_ (0-based indices): [4 5 1 2]
n_support_: [2 2]

Those indices are the support vectors: the active points on the two dotted lines. For a hard-margin fit, $y_i f(x_i)\ge 1$ at every training point, with equality only on this subset. The printed margin $2/\|\beta\|$ is the width between the dotted lines.

Non-separable data and the role of $C$¶

Add a red point at $(X_1,X_2)=(3,1)$. The data are no longer linearly separable.

In [17]:
toy2 = pd.DataFrame({
    "X1": [3, 2, 4, 1, 2, 4, 4, 3],
    "X2": [4, 2, 4, 4, 1, 3, 1, 1],
    "Y":  ["red", "red", "red", "red", "blue", "blue", "blue", "red"],
})
X2 = toy2[["X1", "X2"]].to_numpy()
y2 = toy2["Y"].to_numpy()
toy2
Out[17]:
X1 X2 Y
0 3 4 red
1 2 2 red
2 4 4 red
3 1 4 red
4 2 1 blue
5 4 3 blue
6 4 1 blue
7 3 1 red

A finite C is now required. Large $C$ tries to avoid slack (narrower margin, fewer violations). Small $C$ allows more violations and a wider margin. The plotted intercept is again intercept_ from the solver.

In [18]:
def plot_linear_svm(ax, fit, X, y, title):
    ax.scatter(X[:, 0], X[:, 1], c=y, s=60)
    ax.scatter(
        X[fit.support_, 0], X[fit.support_, 1],
        facecolors="none", edgecolors="black", s=140, linewidths=1.2,
    )
    xs = np.linspace(0, 5, 200)
    xg, yg = np.meshgrid(xs, xs)
    Z = fit.decision_function(np.column_stack([xg.ravel(), yg.ravel()])).reshape(xg.shape)
    ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=2)
    ax.contour(xg, yg, Z, levels=[-1, 1], colors="black", linestyles="dotted")
    ax.set_xlim(0, 5)
    ax.set_ylim(0, 5)
    ax.set_xlabel("X1")
    ax.set_ylabel("X2")
    ax.set_title(title)
    ax.set_aspect("equal")

svm_hiC = SVC(kernel="linear", C=10).fit(X2, y2)
svm_loC = SVC(kernel="linear", C=0.5).fit(X2, y2)

fig, axes = plt.subplots(1, 2, figsize=(8, 4.2))
plot_linear_svm(axes[0], svm_hiC, X2, y2,
                f"C = 10,  nSV = {svm_hiC.n_support_.sum()}")
plot_linear_svm(axes[1], svm_loC, X2, y2,
                f"C = 0.5,  nSV = {svm_loC.n_support_.sum()}")
plt.tight_layout()
plt.show()
No description has been provided for this image

The extra red point is expensive when $C=10$: the boundary moves toward it and the margin shrinks. When $C=0.5$ that point is tolerated as slack, more points become support vectors, and the margin widens. This is an estimation statement, not just a geometric one. Large $C$ is closer to empirical 0-1 risk (fit the training labels, even the noisy ones). Small $C$ is more regularization (a smaller $\|\beta\|$, a wider margin, more slack). The left panel weaves toward the extra point; the right panel is willing to ignore it.

Linear SVM versus logistic regression¶

On the same eight points, fit a linear logistic regression. Both methods give a line; they do not use the data in the same way.

In [19]:
# code red as 1 so the logistic contour is P(red | X)
y01 = (toy2["Y"] == "red").astype(int).to_numpy()
glm = LogisticRegression(C=np.inf, solver="lbfgs").fit(X2, y01)

fig, ax = plt.subplots(figsize=(5.2, 5.2))
ax.scatter(X2[:, 0], X2[:, 1], c=toy2["Y"], s=60)
ax.scatter(
    X2[svm_hiC.support_, 0], X2[svm_hiC.support_, 1],
    facecolors="none", edgecolors="black", s=140, linewidths=1.2,
)

xs = np.linspace(0, 5, 200)
xg, yg = np.meshgrid(xs, xs)
grid = np.column_stack([xg.ravel(), yg.ravel()])
Z_svm = svm_hiC.decision_function(grid).reshape(xg.shape)
Z_glm = glm.decision_function(grid).reshape(xg.shape)
ax.contour(xg, yg, Z_svm, levels=[0], colors="black", linewidths=2)
ax.contour(xg, yg, Z_glm, levels=[0], colors="darkorange", linewidths=2, linestyles="--")
ax.set_xlim(0, 5)
ax.set_ylim(0, 5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_title("Linear SVM (C = 10) vs logistic")
ax.set_aspect("equal")
ax.plot([], [], color="black", label="SVM")
ax.plot([], [], color="darkorange", linestyle="--", label="Logistic")
ax.scatter([], [], facecolors="none", edgecolors="black", s=80, label="SVM support vectors")
ax.legend(loc="lower right", frameon=False)
plt.show()
No description has been provided for this image

Logistic regression is a likelihood for $P(Y=1\mid X)$. Every point contributes to the score equations. The SVM with hinge loss ignores points that already lie beyond the margin; only the support vectors (circled) determine the black line. The two boundaries need not agree, even when both are linear. Logistic regression still produces $\widehat{P}(Y=1\mid X)$; the SVM does not.

Nonlinear SVM: linear kernel, radial kernel, Bayes rule¶

The R lab uses the ESL mixture data from the mda package. That exact object is not in Python, so we generate a close analogue: a two-class Gaussian mixture of the same kind as in ESL, with a known Bayes boundary $P(Y=1\mid X)=1/2$.

A linear SVM can only draw a line. A radial kernel can bend.

In [20]:
rng = np.random.default_rng(1)

# ESL-style mixture: 10 class-specific centers, then N(center, I/5)
n_cent = 10
centers1 = rng.normal(loc=[1.0, 0.0], scale=1.0, size=(n_cent, 2))
centers0 = rng.normal(loc=[0.0, 1.0], scale=1.0, size=(n_cent, 2))

def sample_class(centers, n):
    idx = rng.integers(0, len(centers), size=n)
    return rng.normal(loc=centers[idx], scale=np.sqrt(0.2))

n = 100
X_mix = np.vstack([sample_class(centers1, n), sample_class(centers0, n)])
y_mix = np.array([1] * n + [0] * n)

def bayes_prob(grid, centers1, centers0, var=0.2):
    # equal class priors, equal mixture weights
    def mix_dens(grid, centers):
        d = 0.0
        for c in centers:
            diff = grid - c
            d += np.exp(-0.5 * np.sum(diff**2, axis=1) / var)
        const = 1.0 / (2 * np.pi * var)
        return const * d / len(centers)
    p1 = mix_dens(grid, centers1)
    p0 = mix_dens(grid, centers0)
    return p1 / (p1 + p0)

px1 = np.linspace(X_mix[:, 0].min() - 0.5, X_mix[:, 0].max() + 0.5, 80)
px2 = np.linspace(X_mix[:, 1].min() - 0.5, X_mix[:, 1].max() + 0.5, 80)
xg, yg = np.meshgrid(px1, px2)
grid = np.column_stack([xg.ravel(), yg.ravel()])
prob = bayes_prob(grid, centers1, centers0).reshape(xg.shape)

svm_lin = SVC(kernel="linear", C=10).fit(X_mix, y_mix)
svm_rad = SVC(kernel="rbf", C=5, gamma="scale").fit(X_mix, y_mix)

def plot_mixture(ax, fit, title):
    Z = fit.decision_function(grid).reshape(xg.shape)
    yhat = fit.predict(grid).reshape(xg.shape)
    ax.contourf(xg, yg, yhat, levels=[-0.5, 0.5, 1.5], colors=["#5F9EA0", "#FFE4C4"], alpha=0.35)
    ax.scatter(X_mix[:, 0], X_mix[:, 1],
               c=np.where(y_mix == 1, "darkorange", "deepskyblue"), s=22, zorder=3)
    ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=2)
    ax.contour(xg, yg, prob, levels=[0.5], colors="red", linewidths=2, linestyles="--")
    ax.set_xlabel("x1")
    ax.set_ylabel("x2")
    ax.set_title(title)

fig, axes = plt.subplots(1, 2, figsize=(8, 4.2))
plot_mixture(axes[0], svm_lin, "Linear kernel, C = 10")
plot_mixture(axes[1], svm_rad, "Radial kernel, C = 5")
plt.tight_layout()
plt.show()
No description has been provided for this image

The red dashed curve is the Bayes boundary. The linear kernel cannot track it. The radial kernel can, but only for a reasonable $C$ (and kernel width). That is the point of this figure: the kernel changes the class of boundaries, $C$ changes how hard we fit within that class.

Tuning $C$ by cross-validation¶

$C$ is a regularization parameter. We choose it by cross-validation, not by looking at the training plot. GridSearchCV plays the role of e1071::tune(). The radial kernel also has a width parameter gamma (large gamma is a narrow kernel). In sklearn the RBF kernel is $K(x,z)=\exp(-\gamma\|x-z\|^2)$.

In [21]:
param_lin = {"C": 2.0 ** np.arange(-3, 6)}
tune_lin = GridSearchCV(
    SVC(kernel="linear"),
    param_lin,
    cv=5,
    scoring="accuracy",
    return_train_score=False,
)
tune_lin.fit(X_mix, y_mix)
print("best parameters (linear):", tune_lin.best_params_)
print("best CV accuracy (linear):", tune_lin.best_score_)

Cs = [p["C"] for p in tune_lin.cv_results_["params"]]
mean_err = 1 - tune_lin.cv_results_["mean_test_score"]

fig, ax = plt.subplots(figsize=(6, 4.5))
ax.plot(Cs, mean_err, marker="o")
ax.set_xscale("log", base=2)
ax.set_xlabel("C")
ax.set_ylabel("CV error")
ax.set_title("Linear SVM: cross-validation error vs C")
plt.show()
best parameters (linear): {'C': np.float64(0.125)}
best CV accuracy (linear): 0.775
No description has been provided for this image
In [22]:
param_rad = {"C": [0.5, 1, 5, 10], "gamma": [0.5, 1, 2]}
tune_rad = GridSearchCV(
    SVC(kernel="rbf"),
    param_rad,
    cv=5,
    scoring="accuracy",
)
tune_rad.fit(X_mix, y_mix)
print("best parameters (radial):", tune_rad.best_params_)
print("best CV accuracy (radial):", tune_rad.best_score_)
best parameters (radial): {'C': 10, 'gamma': 0.5}
best CV accuracy (radial): 0.8099999999999999

Read the linear plot as you would a ridge CV curve: very small $C$ underfits (wide margin, too much slack), very large $C$ starts to overfit the training labels. The “best” $C$ is the one with the smallest estimated prediction error, not the prettiest training margin.

GridSearchCV is the analogue of caret::train() / e1071::tune(). What matters is the CV curve, not the wrapper.