9.5b SVM Examples in Python¶
This notebook is the Python version of the lab SVM Examples in R in the lecture notes. The code shows how the objects from the notes — $\beta$, $C$, support vectors, kernels — appear in a fitted SVM.
The calculations use sklearn.svm.SVC. Its argument C is the $C$ of the notes (the same quantity that e1071::svm() calls cost). We do not scale the toy examples, so that coefficients and plots stay in the original units of $X$. In applications one would usually scale the features.
Three comparisons:
a hard margin versus a soft margin (what $C$ does);
a linear SVM versus logistic regression on the same points (who influences the fit);
a linear kernel versus a radial kernel, with a Bayes boundary as a reference.
Linear separable case: maximal margin classifier¶
We start with a small maximal-margin example (adapted from an ISLR exercise).
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
plt.rcParams["figure.dpi"] = 120
toy = pd.DataFrame({
"X1": [3, 2, 4, 1, 2, 4, 4],
"X2": [4, 2, 4, 4, 1, 3, 1],
"Y": ["red", "red", "red", "red", "blue", "blue", "blue"],
})
toy
| X1 | X2 | Y | |
|---|---|---|---|
| 0 | 3 | 4 | red |
| 1 | 2 | 2 | red |
| 2 | 4 | 4 | red |
| 3 | 1 | 4 | red |
| 4 | 2 | 1 | blue |
| 5 | 4 | 3 | blue |
| 6 | 4 | 1 | blue |
A scatterplot of the data:
fig, ax = plt.subplots(figsize=(5, 5))
ax.scatter(toy["X1"], toy["X2"], c=toy["Y"], s=60)
ax.set_xlim(0.5, 4.5)
ax.set_ylim(0.5, 4.5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_aspect("equal")
plt.show()
The linear separating hyperplane has the form
$$ \beta_0 + \beta_1 X_1 + \beta_2 X_2 = 0. $$
We compute it from a hard-margin SVM fit by taking a very large C:
X_toy = toy[["X1", "X2"]].to_numpy()
y_toy = toy["Y"].to_numpy()
svm_toy = SVC(kernel="linear", C=1e5)
svm_toy.fit(X_toy, y_toy)
# sklearn stores beta as coef_ and beta0 as intercept_
beta = svm_toy.coef_.ravel()
beta0 = float(svm_toy.intercept_[0])
beta, beta0
(array([-1.99941406, 1.99941406]), 0.9997070312500023)
So the fitted hyperplane is $\hat\beta_0 + \hat\beta_1 X_1 + \hat\beta_2 X_2 = 0$. The exact numbers can differ in sign from the R lab because sklearn codes the two classes in a fixed order; the line is the same.
The decision boundary in slope–intercept form and the margin width $2/\|\beta\|$:
slope = -beta[0] / beta[1]
intercept = -beta0 / beta[1]
margin = 2 / np.linalg.norm(beta)
slope, intercept, margin
(np.float64(1.0000000000000004), np.float64(-0.5000000000000011), np.float64(0.7073140020855958))
A new point $x^*$ is assigned to a class according to $\operatorname{sign}(f(x^*))$. The solid line below is the separating hyperplane; the dotted lines are the margins.
The fit used a large C and no scaling, so this is, for practical purposes, the hard-margin solution in the original coordinates. In sklearn, support_ holds the (0-based) indices of the support vectors and n_support_ is the count by class. The intercept is intercept_, not the hard-margin midpoint formula from the notes.
xx = np.linspace(0.5, 4.5, 200)
fig, ax = plt.subplots(figsize=(5, 5))
xg, yg = np.meshgrid(xx, xx)
grid = np.column_stack([xg.ravel(), yg.ravel()])
Z = svm_toy.decision_function(grid).reshape(xg.shape)
ax.contourf(xg, yg, Z, levels=[-1e9, 0, 1e9], colors=["#F7E7E7", "#E6F2FA"], alpha=0.8)
ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=1.5)
ax.contour(xg, yg, Z, levels=[-1, 1], colors="gray", linestyles="dotted")
ax.scatter(toy["X1"], toy["X2"], c=toy["Y"], s=60, zorder=3)
ax.scatter(
X_toy[svm_toy.support_, 0],
X_toy[svm_toy.support_, 1],
facecolors="none", edgecolors="black", s=140, linewidths=1.2, zorder=4,
)
ax.set_xlim(0.5, 4.5)
ax.set_ylim(0.5, 4.5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_aspect("equal")
plt.show()
print("support_ (0-based indices):", svm_toy.support_)
print("n_support_:", svm_toy.n_support_)
support_ (0-based indices): [4 5 1 2] n_support_: [2 2]
Those indices are the support vectors: the active points on the two dotted lines. For a hard-margin fit, $y_i f(x_i)\ge 1$ at every training point, with equality only on this subset. The printed margin $2/\|\beta\|$ is the width between the dotted lines.
Non-separable data and the role of $C$¶
Add a red point at $(X_1,X_2)=(3,1)$. The data are no longer linearly separable.
toy2 = pd.DataFrame({
"X1": [3, 2, 4, 1, 2, 4, 4, 3],
"X2": [4, 2, 4, 4, 1, 3, 1, 1],
"Y": ["red", "red", "red", "red", "blue", "blue", "blue", "red"],
})
X2 = toy2[["X1", "X2"]].to_numpy()
y2 = toy2["Y"].to_numpy()
toy2
| X1 | X2 | Y | |
|---|---|---|---|
| 0 | 3 | 4 | red |
| 1 | 2 | 2 | red |
| 2 | 4 | 4 | red |
| 3 | 1 | 4 | red |
| 4 | 2 | 1 | blue |
| 5 | 4 | 3 | blue |
| 6 | 4 | 1 | blue |
| 7 | 3 | 1 | red |
A finite C is now required. Large $C$ tries to avoid slack (narrower margin, fewer violations). Small $C$ allows more violations and a wider margin. The plotted intercept is again intercept_ from the solver.
def plot_linear_svm(ax, fit, X, y, title):
ax.scatter(X[:, 0], X[:, 1], c=y, s=60)
ax.scatter(
X[fit.support_, 0], X[fit.support_, 1],
facecolors="none", edgecolors="black", s=140, linewidths=1.2,
)
xs = np.linspace(0, 5, 200)
xg, yg = np.meshgrid(xs, xs)
Z = fit.decision_function(np.column_stack([xg.ravel(), yg.ravel()])).reshape(xg.shape)
ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=2)
ax.contour(xg, yg, Z, levels=[-1, 1], colors="black", linestyles="dotted")
ax.set_xlim(0, 5)
ax.set_ylim(0, 5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_title(title)
ax.set_aspect("equal")
svm_hiC = SVC(kernel="linear", C=10).fit(X2, y2)
svm_loC = SVC(kernel="linear", C=0.5).fit(X2, y2)
fig, axes = plt.subplots(1, 2, figsize=(8, 4.2))
plot_linear_svm(axes[0], svm_hiC, X2, y2,
f"C = 10, nSV = {svm_hiC.n_support_.sum()}")
plot_linear_svm(axes[1], svm_loC, X2, y2,
f"C = 0.5, nSV = {svm_loC.n_support_.sum()}")
plt.tight_layout()
plt.show()
The extra red point is expensive when $C=10$: the boundary moves toward it and the margin shrinks. When $C=0.5$ that point is tolerated as slack, more points become support vectors, and the margin widens. This is an estimation statement, not just a geometric one. Large $C$ is closer to empirical 0-1 risk (fit the training labels, even the noisy ones). Small $C$ is more regularization (a smaller $\|\beta\|$, a wider margin, more slack). The left panel weaves toward the extra point; the right panel is willing to ignore it.
Linear SVM versus logistic regression¶
On the same eight points, fit a linear logistic regression. Both methods give a line; they do not use the data in the same way.
# code red as 1 so the logistic contour is P(red | X)
y01 = (toy2["Y"] == "red").astype(int).to_numpy()
glm = LogisticRegression(C=np.inf, solver="lbfgs").fit(X2, y01)
fig, ax = plt.subplots(figsize=(5.2, 5.2))
ax.scatter(X2[:, 0], X2[:, 1], c=toy2["Y"], s=60)
ax.scatter(
X2[svm_hiC.support_, 0], X2[svm_hiC.support_, 1],
facecolors="none", edgecolors="black", s=140, linewidths=1.2,
)
xs = np.linspace(0, 5, 200)
xg, yg = np.meshgrid(xs, xs)
grid = np.column_stack([xg.ravel(), yg.ravel()])
Z_svm = svm_hiC.decision_function(grid).reshape(xg.shape)
Z_glm = glm.decision_function(grid).reshape(xg.shape)
ax.contour(xg, yg, Z_svm, levels=[0], colors="black", linewidths=2)
ax.contour(xg, yg, Z_glm, levels=[0], colors="darkorange", linewidths=2, linestyles="--")
ax.set_xlim(0, 5)
ax.set_ylim(0, 5)
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_title("Linear SVM (C = 10) vs logistic")
ax.set_aspect("equal")
ax.plot([], [], color="black", label="SVM")
ax.plot([], [], color="darkorange", linestyle="--", label="Logistic")
ax.scatter([], [], facecolors="none", edgecolors="black", s=80, label="SVM support vectors")
ax.legend(loc="lower right", frameon=False)
plt.show()
Logistic regression is a likelihood for $P(Y=1\mid X)$. Every point contributes to the score equations. The SVM with hinge loss ignores points that already lie beyond the margin; only the support vectors (circled) determine the black line. The two boundaries need not agree, even when both are linear. Logistic regression still produces $\widehat{P}(Y=1\mid X)$; the SVM does not.
Nonlinear SVM: linear kernel, radial kernel, Bayes rule¶
The R lab uses the ESL mixture data from the mda package. That exact object is not in Python, so we generate a close analogue: a two-class Gaussian mixture of the same kind as in ESL, with a known Bayes boundary $P(Y=1\mid X)=1/2$.
A linear SVM can only draw a line. A radial kernel can bend.
rng = np.random.default_rng(1)
# ESL-style mixture: 10 class-specific centers, then N(center, I/5)
n_cent = 10
centers1 = rng.normal(loc=[1.0, 0.0], scale=1.0, size=(n_cent, 2))
centers0 = rng.normal(loc=[0.0, 1.0], scale=1.0, size=(n_cent, 2))
def sample_class(centers, n):
idx = rng.integers(0, len(centers), size=n)
return rng.normal(loc=centers[idx], scale=np.sqrt(0.2))
n = 100
X_mix = np.vstack([sample_class(centers1, n), sample_class(centers0, n)])
y_mix = np.array([1] * n + [0] * n)
def bayes_prob(grid, centers1, centers0, var=0.2):
# equal class priors, equal mixture weights
def mix_dens(grid, centers):
d = 0.0
for c in centers:
diff = grid - c
d += np.exp(-0.5 * np.sum(diff**2, axis=1) / var)
const = 1.0 / (2 * np.pi * var)
return const * d / len(centers)
p1 = mix_dens(grid, centers1)
p0 = mix_dens(grid, centers0)
return p1 / (p1 + p0)
px1 = np.linspace(X_mix[:, 0].min() - 0.5, X_mix[:, 0].max() + 0.5, 80)
px2 = np.linspace(X_mix[:, 1].min() - 0.5, X_mix[:, 1].max() + 0.5, 80)
xg, yg = np.meshgrid(px1, px2)
grid = np.column_stack([xg.ravel(), yg.ravel()])
prob = bayes_prob(grid, centers1, centers0).reshape(xg.shape)
svm_lin = SVC(kernel="linear", C=10).fit(X_mix, y_mix)
svm_rad = SVC(kernel="rbf", C=5, gamma="scale").fit(X_mix, y_mix)
def plot_mixture(ax, fit, title):
Z = fit.decision_function(grid).reshape(xg.shape)
yhat = fit.predict(grid).reshape(xg.shape)
ax.contourf(xg, yg, yhat, levels=[-0.5, 0.5, 1.5], colors=["#5F9EA0", "#FFE4C4"], alpha=0.35)
ax.scatter(X_mix[:, 0], X_mix[:, 1],
c=np.where(y_mix == 1, "darkorange", "deepskyblue"), s=22, zorder=3)
ax.contour(xg, yg, Z, levels=[0], colors="black", linewidths=2)
ax.contour(xg, yg, prob, levels=[0.5], colors="red", linewidths=2, linestyles="--")
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.set_title(title)
fig, axes = plt.subplots(1, 2, figsize=(8, 4.2))
plot_mixture(axes[0], svm_lin, "Linear kernel, C = 10")
plot_mixture(axes[1], svm_rad, "Radial kernel, C = 5")
plt.tight_layout()
plt.show()
The red dashed curve is the Bayes boundary. The linear kernel cannot track it. The radial kernel can, but only for a reasonable $C$ (and kernel width). That is the point of this figure: the kernel changes the class of boundaries, $C$ changes how hard we fit within that class.
Tuning $C$ by cross-validation¶
$C$ is a regularization parameter. We choose it by cross-validation, not by looking at the training plot. GridSearchCV plays the role of e1071::tune(). The radial kernel also has a width parameter gamma (large gamma is a narrow kernel). In sklearn the RBF kernel is $K(x,z)=\exp(-\gamma\|x-z\|^2)$.
param_lin = {"C": 2.0 ** np.arange(-3, 6)}
tune_lin = GridSearchCV(
SVC(kernel="linear"),
param_lin,
cv=5,
scoring="accuracy",
return_train_score=False,
)
tune_lin.fit(X_mix, y_mix)
print("best parameters (linear):", tune_lin.best_params_)
print("best CV accuracy (linear):", tune_lin.best_score_)
Cs = [p["C"] for p in tune_lin.cv_results_["params"]]
mean_err = 1 - tune_lin.cv_results_["mean_test_score"]
fig, ax = plt.subplots(figsize=(6, 4.5))
ax.plot(Cs, mean_err, marker="o")
ax.set_xscale("log", base=2)
ax.set_xlabel("C")
ax.set_ylabel("CV error")
ax.set_title("Linear SVM: cross-validation error vs C")
plt.show()
best parameters (linear): {'C': np.float64(0.125)}
best CV accuracy (linear): 0.775
param_rad = {"C": [0.5, 1, 5, 10], "gamma": [0.5, 1, 2]}
tune_rad = GridSearchCV(
SVC(kernel="rbf"),
param_rad,
cv=5,
scoring="accuracy",
)
tune_rad.fit(X_mix, y_mix)
print("best parameters (radial):", tune_rad.best_params_)
print("best CV accuracy (radial):", tune_rad.best_score_)
best parameters (radial): {'C': 10, 'gamma': 0.5}
best CV accuracy (radial): 0.8099999999999999
Read the linear plot as you would a ridge CV curve: very small $C$ underfits (wide margin, too much slack), very large $C$ starts to overfit the training labels. The “best” $C$ is the one with the smallest estimated prediction error, not the prettiest training margin.
GridSearchCV is the analogue of caret::train() / e1071::tune(). What matters is the CV curve, not the wrapper.