Section 1.5 Python Code¶
We replicate one of the examples from The Elements of Statistical Learning (ESL) to illustrate the differences between the two simplest prediction methods for binary outcomes: linear regression and $k$-nearest neighbors ($k$-NN).
Simulation¶
- The generated data form a binary classification problem with two classes, labeled
0and1. - The features are two-dimensional.
- Class 1 observations are drawn from a bivariate Gaussian distribution with mean $\mu_1$ and variance $\sigma^2$: [ X_{1,1},\ldots,X_{1,n} \sim \mathbb{N}(\mu_1,\sigma^2 I). ]
- Class 0 observations are drawn from a bivariate Gaussian with mean $\mu_0$ and the same variance: [ X_{0,1},\ldots,X_{0,n} \sim \mathbb{N}(\mu_0,\sigma^2 I). ]
- We generate 200 training observations ($n=100$ per class) and assign the corresponding labels.
- We also generate an independent test set of 10,000 observations.
Code¶
Set the simulation parameters:
p = 2 ## No. of parameters
sigma = 1 ## St. Dev for the Normals (common)
mu1 = [1, 0] ## Vector of means for the first Normal
mu0 = [0, 1] ## Vector of means for the second Normal
Generate $n$ i.i.d. (independent and identically distributed) samples from each normal to form the training set:
import numpy as np
# convert to arrays
mu1 = np.asarray(mu1, dtype=float)
mu0 = np.asarray(mu0, dtype=float)
# given/assumed:
n = 100 # training sample size per class
rng = np.random.default_rng(0)
# Gaussian noise: 2n x p i.i.d. N(0, sigma^2)
noise = rng.standard_normal((2*n, p)) * sigma
# Stack n copies of mu1 on top of n copies of mu0 (2n x p)
means = np.vstack([
np.repeat(mu1[None, :], n, axis=0),
np.repeat(mu0[None, :], n, axis=0)
])
# Training data
traindata = noise + means # shape (2n, p)
# Labels: first n are 1's, next n are 0's (to match the R code)
Ytrain = np.concatenate([np.ones(n, dtype=int), np.zeros(n, dtype=int)])
Generate the test set in the same way:
N = 10000
rng = np.random.default_rng(1)
# Gaussian noise: 2N x p i.i.d. N(0, sigma^2)
noise_test = rng.standard_normal((2*N, p)) * sigma
# Stack N copies of mu1 on top of N copies of mu0 (2N x p)
means_test = np.vstack([
np.repeat(mu1[None, :], N, axis=0),
np.repeat(mu0[None, :], N, axis=0)
])
# Test data
testdata = noise_test + means_test
# Labels: first N are 1's, next N are 0's
Ytest = np.concatenate([np.ones(N, dtype=int), np.zeros(N, dtype=int)])
Â
Visualization of the Simulated Data¶
This section also serves as a short review of plotting in R.
We first visualize the training data. In the figure below, Class 1 points are shown in blue and Class 0 points in orange. The two class centers are marked, and a legend is provided.
Â
Plotting using matplotlib¶
import matplotlib.pyplot as plt
# Create an empty plot (equivalent to type="n" in R)
plt.figure(figsize=(10,6))
plt.scatter([], [], label="") # just to set up axes
plt.xlabel("")
plt.ylabel("")
# Plot Class 1 points (first n samples) in blue
plt.scatter(traindata[:n, 0], traindata[:n, 1],
c="blue", marker="o", label="Class 1")
# Plot Class 0 points (next n samples) in orange
plt.scatter(traindata[n:2*n, 0], traindata[n:2*n, 1],
c="orange", marker="o", label="Class 0")
# Add centers: class 1 (green), class 0 (red)
plt.scatter(mu1[0], mu1[1], c="green", marker="x", s=100, label="mu1")
plt.scatter(mu0[0], mu0[1], c="red", marker="x", s=100, label="mu0")
# Legend (just for the two classes, like R’s code)
plt.legend(loc="upper left")
plt.show()
Â
$k$-Nearest Neighbors¶
We now apply $k$-nearest neighbors. The neighborhood sizes used below follow the values suggested in ESL. We also examine the two extreme cases $k=1$ and $k=15$.
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
neighbor_size = [151, 101, 69, 45, 31, 21, 11, 7, 5, 3, 1]
m = len(neighbor_size)
train_err_knn = np.zeros(m)
test_err_knn = np.zeros(m)
for j, k in enumerate(neighbor_size):
# Fit kNN on training data
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(traindata, Ytrain)
# Predictions on training data
Ytrain_pred = knn.predict(traindata)
train_err_knn[j] = np.mean(Ytrain != Ytrain_pred)
# Predictions on testing data
Ytest_pred = knn.predict(testdata)
test_err_knn[j] = np.mean(Ytest != Ytest_pred)
# Put results together (like cbind in R)
results = np.column_stack((train_err_knn, test_err_knn))
print("Train Error | Test Error")
print(results)
Train Error | Test Error [[0.235 0.24 ] [0.235 0.24415] [0.245 0.2444 ] [0.245 0.24595] [0.22 0.24575] [0.225 0.24625] [0.24 0.25115] [0.215 0.25835] [0.23 0.2587 ] [0.19 0.2725 ] [0. 0.32185]]
Â
Choosing $k$ by 5-Fold Cross-Validation¶
A systematic way to select the neighborhood size $k$ is 5-fold cross-validation. For each candidate $k$ we compute the average prediction error across the five folds and choose the value that minimizes this CV error. In the code below the outer loop iterates over the five folds and the inner loop iterates over the candidate values of $k$. In each iteration four folds are used for training and the remaining fold is used for evaluation.
# assumes traindata (shape (2*n, p)), Ytrain (shape (2*n,)),
# testdata (shape (2*N, p)), Ytest (shape (2*N,)) already exist
# and neighbor_size is defined as in your code
# e.g. neighbor_size = [151, 101, 69, 45, 31, 21, 11, 7, 5, 3, 1]
n_total = traindata.shape[0] # should be 2*n
assert n_total % 5 == 0, "Code expects folds of equal size."
fold_size = n_total // 5
# Make a random permutation of indices 0..(2n-1) (R used 1..2n then 1-based slicing)
rng = np.random.default_rng(0) # set seed if you want reproducibility
idx = rng.permutation(n_total)
# Fold boundaries like R's: c(0, 40, 80, 120, 160, 200) when 2n=200
fold_bounds = np.arange(0, n_total + 1, fold_size)
m = len(neighbor_size)
cv_error_counts = np.zeros(m, dtype=int) # accumulate misclassification counts over all folds
# 5-fold CV
for i in range(5):
val_idx = idx[fold_bounds[i]:fold_bounds[i+1]]
train_idx = np.setdiff1d(idx, val_idx, assume_unique=True)
X_tr, y_tr = traindata[train_idx], Ytrain[train_idx]
X_val, y_val = traindata[val_idx], Ytrain[val_idx]
for j, k in enumerate(neighbor_size):
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(X_tr, y_tr)
y_val_pred = knn.predict(X_val)
cv_error_counts[j] += np.sum(y_val_pred != y_val)
# Choose k with the smallest total CV misclassification count
k_optimal = neighbor_size[int(np.argmin(cv_error_counts))]
cv_error_rate = cv_error_counts / n_total # optional, to view as rates
# print("CV misclassification counts per k:", cv_error_counts)
# print("CV error rates per k:", cv_error_rate)
print("Optimal k (5-fold CV):", k_optimal)
# Train/test errors using the CV-selected k
knn_cv = KNeighborsClassifier(n_neighbors=k_optimal)
knn_cv.fit(traindata, Ytrain)
Ytrain_pred = knn_cv.predict(traindata)
train_err_knn_CV = np.mean(Ytrain_pred != Ytrain)
Ytest_pred = knn_cv.predict(testdata)
test_err_knn_CV = np.mean(Ytest_pred != Ytest)
print("Train error (CV-chosen k):", train_err_knn_CV)
print("Test error (CV-chosen k):", test_err_knn_CV)
Optimal k (5-fold CV): 101 Train error (CV-chosen k): 0.235 Test error (CV-chosen k): 0.24415
 
Linear Regression (Least Squares)¶
We fit a linear regression of the numeric response on the two features and convert the continuous fitted values into class predictions by the rule
$$ \widehat{Y} = \begin{cases} 1, & \text{ if } fitted(Y) > 0.5\\ 0, & \text{ if } fitted(Y) \leq 0.5 \end{cases} $$
Â
from sklearn.linear_model import LinearRegression
from sklearn.metrics import confusion_matrix
# Ytrain is {0,1}, already numeric
# traindata is (2*n, p), testdata is (2*N, p)
# Fit linear regression model
reg = LinearRegression()
reg.fit(traindata, Ytrain)
# Training predictions: fitted values > 0.5 → class 1, else class 0
Ytrain_pred_LS = (reg.predict(traindata) > 0.5).astype(int)
# Testing predictions
Ytest_pred_cont = reg.predict(testdata) # continuous values
Ytest_pred_LS = (Ytest_pred_cont > 0.5).astype(int)
# Cross-tabs (confusion matrices)
train_cm = confusion_matrix(Ytrain, Ytrain_pred_LS, labels=[0,1])
test_cm = confusion_matrix(Ytest, Ytest_pred_LS, labels=[0,1])
train_err_LS = np.mean(Ytrain != Ytrain_pred_LS)
test_err_LS = np.mean(Ytest != Ytest_pred_LS)
print("Train confusion matrix:\n", train_cm)
print("Train error (LS):", train_err_LS)
print("\nTest confusion matrix:\n", test_cm)
print("Test error (LS):", test_err_LS)
# Inspect coefficients (like R’s RegModel$coef)
# print("\nIntercept:", reg.intercept_)
# print("Coefficients:", reg.coef_)
Train confusion matrix: [[75 25] [20 80]] Train error (LS): 0.225 Test confusion matrix: [[7307 2693] [2154 7846]] Test error (LS): 0.24235
# --- Define plot ranges like your R code ---
x_min = np.round(traindata[:, 0].min(), 1) - 0.1
x_max = np.round(traindata[:, 0].max(), 1) + 0.1
y_min = np.round(traindata[:, 1].min(), 1) - 0.1
y_max = np.round(traindata[:, 1].max(), 1) + 0.1
x_range = np.arange(x_min, x_max + 1e-9, 0.1)
y_range = np.arange(y_min, y_max + 1e-9, 0.1)
XX, YY = np.meshgrid(x_range, y_range)
grid_points = np.c_[XX.ravel(), YY.ravel()] # same as expand.grid
# Base scatter (like grid.plot in ggplot)
def base_scatter(ax):
# Ytrain: 1 -> blue, 0 -> orange to match your ggplot colors
colors = np.where(Ytrain == 1, "blue", "orange")
ax.scatter(traindata[:, 0], traindata[:, 1], c=colors, s=20, edgecolor="none")
# Centers
ax.scatter(mu1[0], mu1[1], c="red", marker="^", s=80) # class 1 center
ax.scatter(mu0[0], mu0[1], c="green", marker="^", s=80) # class 0 center
ax.set_xlabel("")
ax.set_ylabel("")
ax.set_xlim(x_min, x_max)
ax.set_ylim(y_min, y_max)
ax.legend(handles=[
plt.Line2D([], [], marker="o", linestyle="", color="blue", label="Class 1"),
plt.Line2D([], [], marker="o", linestyle="", color="orange", label="Class 0")
], loc="lower right")
def plot_knn_decision(k, title):
# Fit kNN on training set
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(traindata, Ytrain)
# Predict on grid
# For a shaded background like ggplot, we can use class predictions on the grid
grid_pred = knn.predict(grid_points).reshape(XX.shape)
# For a crisp decision boundary, use class-1 probability and contour at 0.5
if hasattr(knn, "predict_proba"):
grid_proba1 = knn.predict_proba(grid_points)[:, 1].reshape(XX.shape)
else:
# Fallback (rare for KNN): approximate boundary by cast to float of predictions
grid_proba1 = grid_pred.astype(float)
fig, ax = plt.subplots(figsize=(8, 6))
base_scatter(ax)
# Shaded regions: match colors (0 -> orange, 1 -> blue)
# We'll map 0/1 to 0/1 and use a ListedColormap with the same two colors
from matplotlib.colors import ListedColormap
cmap_bg = ListedColormap(["orange", "blue"])
ax.pcolormesh(XX, YY, grid_pred, shading="auto", alpha=0.15, cmap=cmap_bg)
# Decision contour at prob=0.5 (1 bin/level in ggplot)
CS = ax.contour(XX, YY, grid_proba1, levels=[0.5], colors="black", linewidths=1.0)
ax.clabel(CS, fmt={0.5: "boundary"}, inline=True, fontsize=8)
ax.set_title(f"kNN Classification for k={k}")
plt.show()
# Plots equivalent to your ggplot figures:
plot_knn_decision(k=15, title="kNN Classification for k=15")
plot_knn_decision(k=1, title="kNN Classification for k=1")
Â
Linear-regression classification¶
from matplotlib.colors import ListedColormap
# --- Fit linear regression: Y ~ X1 + X2 (with intercept) ---
reg = LinearRegression()
reg.fit(traindata, Ytrain)
beta0 = reg.intercept_ # corresponds to beta.hat[1] in R (1-based)
beta1, beta2 = reg.coef_ # corresponds to beta.hat[2], beta.hat[3]
# --- Predict on grid (like Reg.yhat = [1, X] %*% beta.hat) ---
grid_yhat = reg.predict(grid_points) # continuous predictions
grid_pred = (grid_yhat > 0.5).astype(int) # class labels as 0/1
grid_pred_img = grid_pred.reshape(XX.shape)
# --- Decision boundary (beta0 + beta1*x + beta2*y = 0.5) ---
# Solve for y: y = -(beta1/beta2) * x + (0.5 - beta0)/beta2
# (only draw if beta2 != 0)
fig, ax = plt.subplots(figsize=(8, 6))
base_scatter(ax)
# Shaded class regions to mirror the R ggplot points background
cmap_bg = ListedColormap(["orange", "blue"])
ax.pcolormesh(XX, YY, grid_pred_img, shading="auto", alpha=0.15, cmap=cmap_bg)
# Decision boundary line
if np.abs(beta2) > 1e-12:
x_line = np.array([ax.get_xlim()[0], ax.get_xlim()[1]])
slope = -beta1 / beta2
intercept = (0.5 - beta0) / beta2
y_line = slope * x_line + intercept
ax.plot(x_line, y_line, color="black", linewidth=1.5)
else:
# Vertical boundary when beta2 ≈ 0: beta0 + beta1*x = 0.5 → x = (0.5 - beta0)/beta1
if np.abs(beta1) > 1e-12:
x_vert = (0.5 - beta0) / beta1
ax.axvline(x_vert, color="black", linewidth=1.5)
ax.set_title("Regression Classification")
plt.show()
Comparing Training and Test Performance¶
The figure below shows training error (blue) and test error (magenta) as functions of model complexity.
- The upper $x$-axis shows the neighborhood size $k$.
- The lower $x$-axis shows the approximate degrees of freedom of the $k$-NN fit ($2n/k$).
- Linear regression (3 parameters) is plotted at 3 degrees of freedom.
- The $k$-NN results that correspond to the value selected by cross-validation are marked with diamonds.
# Degrees of freedom: round((2*n)/k)
df = np.round((2*n) / np.array(neighbor_size)).astype(int)
x = np.arange(1, m+1)
y_min = min(
np.min(test_err_LS),
np.min(train_err_LS),
np.min(test_err_knn),
np.min(train_err_knn),
)
y_max = max(
np.max(test_err_LS),
np.max(train_err_LS),
np.max(test_err_knn),
np.max(train_err_knn),
)
fig, ax = plt.subplots(figsize=(9, 6))
# Empty base (like type="n")
ax.set_xlim(0.5, m + 0.5)
ax.set_ylim(y_min - 0.01, y_max + 0.01)
ax.set_xlabel("Degrees of Freedom")
ax.set_ylabel("Error")
# Bottom axis ticks: degrees of freedom
ax.set_xticks(x)
ax.set_xticklabels(df)
# Top axis: k (neighbor_size)
ax_top = ax.twiny()
ax_top.set_xlim(ax.get_xlim())
ax_top.set_xticks(x)
ax_top.set_xticklabels(neighbor_size)
ax_top.set_xlabel("k (neighbors)")
# kNN curves
ax.plot(x, test_err_knn, color="magenta", linewidth=2, linestyle="-", marker="o", label="kNN Test Error")
ax.plot(x, train_err_knn, color="blue", linewidth=2, linestyle="--", marker="o", label="kNN Train Error")
# Linear regression errors at x=3 (to match your R code)
ax.scatter([3], [train_err_LS], color="blue", marker="^", s=80, label="Regression Training Error")
ax.scatter([3], [test_err_LS], color="magenta", marker="^", s=80, label="Regression Testing Error")
# CV-chosen k marker at its position
pos_opt = x[np.array(neighbor_size) == k_optimal][0] # 1-based position
ax.scatter([pos_opt], [train_err_knn_CV], color="blue", marker="s", s=90, label="kNN Optimal k Training Error")
ax.scatter([pos_opt], [test_err_knn_CV], color="magenta", marker="s", s=90, label="kNN Optimal k Testing Error")
# Legend (grouped, similar to R)
ax.legend(loc="lower left", fontsize=9)
plt.tight_layout()
plt.show()