Section 3.4 Python Code and Analysis¶
The Student Performance Example¶
In this final section we illustrate the practical implementation of the variable-selection and shrinkage methods developed earlier, using a real data example.
The analysis is based on the student-mat.csv data set, which contains information on student achievement in secondary education at two Portuguese schools. The original study is described in:
Cortez, P., & Silva, A. M. G. (2008). Using data mining to predict secondary school student performance. Proceedings of the 5th Annual Future Business Technology Conference.
The data are publicly available from the UCI Machine Learning Repository:
Student Performance data set.
The variables in the data set are the following:
school– student’s school (binary:GP– Gabriel Pereira orMS– Mousinho da Silveira)sex– student’s sex (binary:F– female orM– male)age– student’s age (numeric: from 15 to 22)address– home address type (binary:U– urban orR– rural)famsize– family size (binary:LE3– ≤ 3 orGT3– > 3)Pstatus– parent’s cohabitation status (binary:T– living together orA– apart)Medu– mother’s education (numeric: 0 – none, 1 – primary (4th grade), 2 – 5th–9th grade, 3 – secondary, 4 – higher education)Fedu– father’s education (numeric: same coding asMedu)Mjob– mother’s occupation (nominal:teacher,health,services,at_home,other)Fjob– father’s occupation (nominal: same categories asMjob)reason– reason for choosing the school (nominal:home,reputation,course,other)guardian– student’s guardian (nominal:mother,father,other)traveltime– home-to-school travel time (numeric: 1 – <15 min, 2 – 15–30 min, 3 – 30–60 min, 4 – >1 hour)studytime– weekly study time (numeric: 1 – <2 h, 2 – 2–5 h, 3 – 5–10 h, 4 – >10 h)failures– number of past class failures (numeric: n if 1 ≤ n < 3, else 4)schoolsup– extra educational support (binary:yes/no)famsup– family educational support (binary:yes/no)paid– extra paid classes in the course subject (binary:yes/no)activities– extra-curricular activities (binary:yes/no)nursery– attended nursery school (binary:yes/no)higher– wants to take higher education (binary:yes/no)internet– Internet access at home (binary:yes/no)romantic– in a romantic relationship (binary:yes/no)famrel– quality of family relationships (numeric: 1 – very bad to 5 – excellent)freetime– free time after school (numeric: 1 – very low to 5 – very high)goout– going out with friends (numeric: 1 – very low to 5 – very high)Dalc– workday alcohol consumption (numeric: 1 – very low to 5 – very high)Walc– weekend alcohol consumption (numeric: 1 – very low to 5 – very high)health– current health status (numeric: 1 – very bad to 5 – very good)absences– number of school absences (numeric: 0 to 93)G1– first-period grade (numeric: 0 to 20)G2– second-period grade (numeric: 0 to 20)G3– final grade (numeric: 0 to 20; the response variable)
import pandas as pd
stu_performance0 = pd.read_csv("student-mat.csv", sep=';')
Our goal is to predict the final grade (G3) using the penalized regression methods studied in this chapter. Because standard ridge and lasso do not handle categorical predictors directly (a group-lasso formulation would be required), we restrict attention to the numeric predictors only.
cols_to_drop = [
'school', 'sex', 'address', 'famsize', 'Pstatus',
'Mjob', 'Fjob', 'reason', 'guardian', 'schoolsup',
'famsup', 'paid', 'activities', 'nursery', 'higher',
'internet', 'romantic', 'G1', 'G2'
]
stu_performance = stu_performance0.drop(columns=cols_to_drop) # data frame without categorical variables
# manually renaming response to 'Grades'
stu_performance = stu_performance.rename(columns={'G3': 'Grades'})
n = stu_performance.shape[0] # sample size (number of rows)
p = stu_performance.shape[1] - 1 # number of non-intercept predictors (columns - 1)
stu_performance.head()
| age | Medu | Fedu | traveltime | studytime | failures | famrel | freetime | goout | Dalc | Walc | health | absences | Grades | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 18 | 4 | 4 | 2 | 2 | 0 | 4 | 3 | 4 | 1 | 1 | 3 | 6 | 6 |
| 1 | 17 | 1 | 1 | 1 | 2 | 0 | 5 | 3 | 3 | 1 | 1 | 3 | 4 | 6 |
| 2 | 15 | 1 | 1 | 1 | 2 | 3 | 4 | 3 | 2 | 2 | 3 | 3 | 10 | 10 |
| 3 | 15 | 4 | 2 | 1 | 3 | 0 | 3 | 2 | 2 | 1 | 1 | 5 | 2 | 15 |
| 4 | 16 | 3 | 3 | 1 | 2 | 0 | 4 | 3 | 2 | 1 | 2 | 5 | 4 | 10 |
We begin by fitting the full linear model that includes every numeric predictor:
import statsmodels.formula.api as smf
# due to error with patsy ~. formula, workaround by building the formula "explicitly"
predictors = stu_performance.columns.drop("Grades")
formula = "Grades ~ " + " + ".join(predictors)
performance_full = smf.ols(formula=formula, data=stu_performance).fit()
performance_full.summary()
| Dep. Variable: | Grades | R-squared: | 0.183 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.155 |
| Method: | Least Squares | F-statistic: | 6.564 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 2.53e-11 |
| Time: | 12:15:49 | Log-Likelihood: | -1121.3 |
| No. Observations: | 395 | AIC: | 2271. |
| Df Residuals: | 381 | BIC: | 2326. |
| Df Model: | 13 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 13.5370 | 3.337 | 4.056 | 0.000 | 6.975 | 20.099 |
| age | -0.2562 | 0.179 | -1.433 | 0.153 | -0.608 | 0.095 |
| Medu | 0.5849 | 0.257 | 2.280 | 0.023 | 0.081 | 1.089 |
| Fedu | -0.0913 | 0.255 | -0.359 | 0.720 | -0.592 | 0.409 |
| traveltime | -0.4018 | 0.314 | -1.278 | 0.202 | -1.020 | 0.216 |
| studytime | 0.2658 | 0.269 | 0.988 | 0.324 | -0.263 | 0.795 |
| failures | -1.8315 | 0.312 | -5.870 | 0.000 | -2.445 | -1.218 |
| famrel | 0.2802 | 0.245 | 1.142 | 0.254 | -0.202 | 0.763 |
| freetime | 0.3414 | 0.231 | 1.478 | 0.140 | -0.113 | 0.796 |
| goout | -0.6264 | 0.222 | -2.824 | 0.005 | -1.062 | -0.190 |
| Dalc | -0.1372 | 0.320 | -0.428 | 0.669 | -0.767 | 0.493 |
| Walc | 0.3531 | 0.238 | 1.482 | 0.139 | -0.116 | 0.822 |
| health | -0.1743 | 0.156 | -1.116 | 0.265 | -0.481 | 0.133 |
| absences | 0.0320 | 0.028 | 1.158 | 0.247 | -0.022 | 0.086 |
| Omnibus: | 28.620 | Durbin-Watson: | 2.062 |
|---|---|---|---|
| Prob(Omnibus): | 0.000 | Jarque-Bera (JB): | 32.727 |
| Skew: | -0.677 | Prob(JB): | 7.82e-08 |
| Kurtosis: | 3.391 | Cond. No. | 314. |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
A simple (but inefficient) approach to model selection is to compare the full model with a reduced model that retains only the apparently significant variables, using a partial F-test:
import statsmodels.api as sm
from statsmodels.stats.anova import anova_lm
# first we fit a reduced model
formula_reduced = 'Grades ~ failures + goout + Medu'
performance_red = smf.ols(formula=formula_reduced, data=stu_performance).fit()
anova_results = anova_lm(performance_red, performance_full)
print(anova_results)
df_resid ssr df_diff ss_diff F Pr(>F) 0 391.0 6956.920171 0.0 NaN NaN NaN 1 381.0 6756.704779 10.0 200.215392 1.128983 0.338977
In this case, we fail to reject the null so the reduced model is preferred. However, this approach is not efficient nor optimal. Therefore, we use algorithms to search the space of models and a penalty to penalize the inclusion of terms that are not as useful.
Subset Selection via Level-wise algorithms¶
import itertools
import statsmodels.api as sm
import pandas as pd
import numpy as np
# Data
X = stu_performance.drop(columns="Grades")
y = stu_performance["Grades"]
# Full model MSE for Mallows' Cp
sigma2_full = performance_full.mse_resid
# max subset size
nvmax = X.shape[1]
# we will store the best model per subset size
best_per_size = []
for k in range(1, nvmax + 1):
best_model = None
best_rsq = -np.inf
# Loop over all combinations of size k using itertools combinations
for combo in itertools.combinations(X.columns, k):
X_subset = sm.add_constant(X[list(combo)])
model = sm.OLS(y, X_subset).fit()
if model.rsquared > best_rsq:
best_rsq = model.rsquared
best_model = {
"k": k,
"predictors": combo,
"rsq": model.rsquared,
"adj_rsq": model.rsquared_adj,
"bic": model.bic,
"aic": model.aic,
"rss": model.ssr
}
# Compute Mallows' Cp
best_model["cp"] = best_model["rss"] / sigma2_full - (len(y) - 2*len(best_model["predictors"]))
best_per_size.append(best_model)
rs = pd.DataFrame(best_per_size)
The `rs` object above contains all the different criteria for all the models that have been evaluated.
Below we extract the $R^2$, Adjusted $R^2$, $AIC$, $BIC$ and $C_p$-Mallows criteria for the 13 models that have been evaluated:
rs
| k | predictors | rsq | adj_rsq | bic | aic | rss | cp | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 | (failures,) | 0.129899 | 0.127685 | 2279.346318 | 2271.388546 | 7195.656554 | 12.751803 |
| 1 | 2 | (Medu, failures) | 0.148313 | 0.143968 | 2276.875838 | 2264.939180 | 7043.370273 | 6.164618 |
| 2 | 3 | (Medu, failures, goout) | 0.158767 | 0.152313 | 2277.976507 | 2262.060964 | 6956.920171 | 3.289832 |
| 3 | 4 | (Medu, failures, freetime, goout) | 0.163370 | 0.154789 | 2281.788229 | 2261.893800 | 6918.855608 | 3.143431 |
| 4 | 5 | (Medu, traveltime, failures, freetime, goout) | 0.166832 | 0.156123 | 2286.128964 | 2262.255649 | 6890.221033 | 3.528772 |
| 5 | 6 | (Medu, traveltime, failures, freetime, goout, ... | 0.169663 | 0.156823 | 2290.763350 | 2262.911150 | 6866.807998 | 4.208548 |
| 6 | 7 | (age, Medu, traveltime, failures, freetime, go... | 0.172408 | 0.157439 | 2295.434493 | 2263.603407 | 6844.111358 | 4.928720 |
| 7 | 8 | (age, Medu, traveltime, failures, freetime, go... | 0.174691 | 0.157586 | 2300.322023 | 2264.512051 | 6825.227682 | 5.863899 |
| 8 | 9 | (age, Medu, traveltime, failures, famrel, free... | 0.177358 | 0.158127 | 2305.022777 | 2265.233919 | 6803.178460 | 6.620578 |
| 9 | 10 | (age, Medu, traveltime, failures, famrel, free... | 0.180038 | 0.158685 | 2309.712285 | 2265.944542 | 6781.007415 | 7.370388 |
| 10 | 11 | (age, Medu, traveltime, studytime, failures, f... | 0.182315 | 0.158831 | 2314.592990 | 2266.846361 | 6762.181008 | 8.308796 |
| 11 | 12 | (age, Medu, traveltime, studytime, failures, f... | 0.182701 | 0.157027 | 2320.385250 | 2268.659736 | 6758.986840 | 10.128682 |
| 12 | 13 | (age, Medu, Fedu, traveltime, studytime, failu... | 0.182977 | 0.155100 | 2326.230748 | 2270.526348 | 6756.704779 | 12.000000 |
Going by $r^2$, the best model is the 13th as expected. By adjusted $r^2$ it is the 11th; by CP it is the 4th, and by BIC the best model is the 2nd one.
We can identify which variables are included by looking at the predictors column
print(rs["predictors"].iloc[1])
('Medu', 'failures')
For example, the model with two variables contains only failures, Medu and the intercept. The model with only one feature contains onlyfailures, and the intercept.
We can also compute AIC and BIC "by hand" for the sequence of models:
rss = rs['rss']
msize = np.arange(1, p + 1)
Aic = n * np.log(rss / n) + 2 * msize
Bic = n * np.log(rss / n) + msize * np.log(n)
Aic
0 1148.427105 1 1141.977739 2 1139.099523 3 1138.932359 4 1139.294208 5 1139.949709 6 1140.641966 7 1141.550610 8 1142.272478 9 1142.983101 10 1143.884920 11 1145.698294 12 1147.564906 Name: rss, dtype: float64
AIC chooses the 4th model.
Bic
0 1152.405991 1 1149.935511 2 1151.036180 3 1154.847902 4 1159.188637 5 1163.823023 6 1168.494166 7 1173.381696 8 1178.082450 9 1182.771958 10 1187.652663 11 1193.444923 12 1199.290421 Name: rss, dtype: float64
As we saw before, BIC selects the 2nd model.
We can also plot the results for the various models. So, we have
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 2)
axes[0, 0].plot(msize, rs['adj_rsq'])
axes[0, 0].set_xlabel("No. of Parameters")
axes[0, 0].set_ylabel("Adjusted Rsquare")
axes[0, 1].plot(msize, rs['cp'])
axes[0, 1].set_xlabel("No. of Parameters")
axes[0, 1].set_ylabel("Mallow's Cp")
axes[1, 0].plot(msize, Aic)
axes[1, 0].set_xlabel("No. of Parameters")
axes[1, 0].set_ylabel("AIC")
# R: plot(msize, Bic, xlab="No. of Parameters", ylab = "BIC")
axes[1, 1].plot(msize, Bic)
axes[1, 1].set_xlabel("No. of Parameters")
axes[1, 1].set_ylabel("BIC")
plt.tight_layout() # cleans it up
plt.show()
As is typical, AIC selects a larger model than BIC. Although the BIC-optimal model does not attain the absolute minimum AIC, its AIC value is very close to the optimum.
best_k_aic = Aic.idxmin()
best_k_bic = Bic.idxmin()
comparison_df = pd.DataFrame({
f'AIC-Best ({best_k_aic} features)': rs.loc[best_k_aic],
f'BIC-Best ({best_k_bic} features)': rs.loc[best_k_bic]
})
comparison_df
| AIC-Best (3 features) | BIC-Best (1 features) | |
|---|---|---|
| k | 4 | 2 |
| predictors | (Medu, failures, freetime, goout) | (Medu, failures) |
| rsq | 0.16337 | 0.148313 |
| adj_rsq | 0.154789 | 0.143968 |
| bic | 2281.788229 | 2276.875838 |
| aic | 2261.8938 | 2264.93918 |
| rss | 6918.855608 | 7043.370273 |
| cp | 3.143431 | 6.164618 |
Note that the manually computed AIC and BIC values are different from those calculated using the sm package. But the difference is only a constant and so the same models are selected.
What are the 2nd and 3rd best models in terms of AIC/BIC?
This cannot be answered by looking at the AIC/BIC plots shown above. Instead, we need to run the following code.
all_models = []
for k in range(1, p + 1):
for combo in itertools.combinations(X.columns, k):
feature_set = list(combo)
model = sm.OLS(y, sm.add_constant(X[feature_set])).fit()
all_models.append({
"num_features": len(feature_set),
"features": feature_set,
"rss": model.ssr
})
all_models_df = pd.DataFrame(all_models)
# Get the top 3 models for each subset size, based on lowest RSS
nbest = 3
top_nbest_id = all_models_df.groupby('num_features')['rss'].nsmallest(nbest).index.get_level_values(1)
rs_1_df = all_models_df.loc[top_nbest_id]
# Replicate msize.1
msize_1 = rs_1_df['num_features']
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
Aic_1 = n * np.log(rs_1_df['rss'] / n) + 2 * msize_1
Bic_1 = n * np.log(rs_1_df['rss'] / n) + msize_1 * np.log(n)
axes[0].scatter(msize_1, Aic_1)
axes[0].set_xlabel("No. of Parameters")
axes[0].set_ylabel("AIC")
axes[1].scatter(msize_1, Bic_1)
axes[1].set_xlabel("No. of Parameters")
axes[1].set_ylabel("BIC")
plt.show()
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].scatter(msize_1, Aic_1)
axes[0].set_xlabel("No. of Parameters")
axes[0].set_ylabel("AIC")
axes[0].set_ylim(1135, 1152)
axes[1].scatter(msize_1, Bic_1)
axes[1].set_xlabel("No. of Parameters")
axes[1].set_ylabel("BIC")
axes[1].set_ylim(1140, 1230)
plt.show()
which_df_1 = pd.DataFrame(False, index=rs_1_df.index, columns=X.columns)
for index, row in rs_1_df.iterrows():
which_df_1.loc[index, row['features']] = True
# Add AIC/BIC to the df to sort
which_df_1['Aic'] = Aic_1
which_df_1['Bic'] = Bic_1
print("--- Top 3 models by AIC ---")
print(which_df_1.sort_values('Aic').head(3).drop(columns=['Aic', 'Bic']))
print("\n--- Top 3 models by BIC ---")
print(which_df_1.sort_values('Bic').head(3).drop(columns=['Aic', 'Bic']))
--- Top 3 models by AIC ---
age Medu Fedu traveltime studytime failures famrel freetime \
712 False True False False False True False True
186 False True False False False True False False
1741 False True False True False True False True
goout Dalc Walc health absences
712 True False False False False
186 True False False False False
1741 True False False False False
--- Top 3 models by BIC ---
age Medu Fedu traveltime studytime failures famrel freetime \
28 False True False False False True False False
186 False True False False False True False False
5 False False False False False True False False
goout Dalc Walc health absences
28 False False False False False
186 True False False False False
5 False False False False False
We can use the stepwise approach to search possible models moving forward, backwards or in both directions (stepwise).
Python does not have a prebuilt function that mimicks the step function R. However, we can mimick this code using a for loop. Note that in the following function the default value for $k =2$, and uses AIC. The criteria can be changed to BIC by changing $k$ appropriately.
import numpy as np
import statsmodels.formula.api as smf
def stepwise_selection(model, k=2): # For BIC, use k=np.log(len(model.model.data.frame))
"""
Performs bidirectional (forward/backward) variable selection based on a criterion.
Takes a fitted statsmodels model as input, mimicking R's step().
"""
df = model.model.data.frame
response = model.model.endog_names
# Get initial predictors from the model, excluding the intercept
initial_predictors = {p for p in model.model.exog_names if p != 'Intercept'}
# Get all possible predictors from the dataframe
all_predictors = {col for col in df.columns if col != response}
included = initial_predictors
excluded = all_predictors - included
while True:
# Define the current best model and its criterion
current_formula = f"{response} ~ {' + '.join(included) or '1'}"
current_model = smf.ols(current_formula, data=df).fit()
# The fix is here: using .df_model instead of .df_modelwc
current_criterion = -2 * current_model.llf + k * current_model.df_model
best_candidate_criterion = current_criterion
best_move = None
# Check backward moves (dropping a variable)
for var in included:
formula = f"{response} ~ {' + '.join(included - {var}) or '1'}"
temp_model = smf.ols(formula, data=df).fit()
criterion = -2 * temp_model.llf + k * temp_model.df_model
if criterion < best_candidate_criterion:
best_candidate_criterion = criterion
best_move = ('drop', var)
# Check forward moves (adding a variable)
for var in excluded:
formula = f"{response} ~ {' + '.join(included | {var}) or '1'}"
temp_model = smf.ols(formula, data=df).fit()
criterion = -2 * temp_model.llf + k * temp_model.df_model
if criterion < best_candidate_criterion:
best_candidate_criterion = criterion
best_move = ('add', var)
# If a better model was found, make the move
if best_move:
move_type, var = best_move
if move_type == 'drop':
included.remove(var)
excluded.add(var)
else: # 'add'
excluded.remove(var)
included.add(var)
else:
# If no move improves the criterion, stop
break
final_formula = f"{response} ~ {' + '.join(included) or '1'}"
return smf.ols(final_formula, data=df).fit()
stepAIC = stepwise_selection(performance_full, k=2)
n = len(stu_performance)
stepBIC = stepwise_selection(performance_full, k=np.log(n))
stepAIC.summary()
| Dep. Variable: | Grades | R-squared: | 0.163 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.155 |
| Method: | Least Squares | F-statistic: | 19.04 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 2.57e-14 |
| Time: | 12:15:59 | Log-Likelihood: | -1125.9 |
| No. Observations: | 395 | AIC: | 2262. |
| Df Residuals: | 390 | BIC: | 2282. |
| Df Model: | 4 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 9.8868 | 0.970 | 10.194 | 0.000 | 7.980 | 11.794 |
| goout | -0.5064 | 0.201 | -2.524 | 0.012 | -0.901 | -0.112 |
| freetime | 0.3254 | 0.222 | 1.465 | 0.144 | -0.111 | 0.762 |
| Medu | 0.6190 | 0.201 | 3.087 | 0.002 | 0.225 | 1.013 |
| failures | -1.9505 | 0.297 | -6.558 | 0.000 | -2.535 | -1.366 |
| Omnibus: | 37.170 | Durbin-Watson: | 2.041 |
|---|---|---|---|
| Prob(Omnibus): | 0.000 | Jarque-Bera (JB): | 45.218 |
| Skew: | -0.769 | Prob(JB): | 1.52e-10 |
| Kurtosis: | 3.618 | Cond. No. | 25.6 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
stepBIC.summary()
| Dep. Variable: | Grades | R-squared: | 0.148 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.144 |
| Method: | Least Squares | F-statistic: | 34.13 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 2.16e-14 |
| Time: | 12:15:59 | Log-Likelihood: | -1129.5 |
| No. Observations: | 395 | AIC: | 2265. |
| Df Residuals: | 392 | BIC: | 2277. |
| Df Model: | 2 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 9.4821 | 0.621 | 15.267 | 0.000 | 8.261 | 10.703 |
| Medu | 0.5845 | 0.201 | 2.911 | 0.004 | 0.190 | 0.979 |
| failures | -2.0168 | 0.296 | -6.824 | 0.000 | -2.598 | -1.436 |
| Omnibus: | 35.775 | Durbin-Watson: | 2.001 |
|---|---|---|---|
| Prob(Omnibus): | 0.000 | Jarque-Bera (JB): | 42.955 |
| Skew: | -0.760 | Prob(JB): | 4.70e-10 |
| Kurtosis: | 3.545 | Cond. No. | 9.63 |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
Notice that all of the variables in stepBIC are strictly included in the variables of stepAIC
stepAIC.model.exog_names
['Intercept', 'goout', 'freetime', 'Medu', 'failures']
stepBIC.model.exog_names
['Intercept', 'Medu', 'failures']
set(stepBIC.model.exog_names).issubset(set(stepAIC.model.exog_names))
True
LASSO and RIDGE Regression¶
SciKitLearn has packages for Lasso and Ridge
from sklearn.linear_model import LassoCV, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X = stu_performance.drop(columns="Grades")
y = stu_performance["Grades"]
We'll first break up the data into train (80%) and test (20%) and create a full model.
#split the data and assign
train_df, test_df = train_test_split(stu_performance, test_size=0.2, random_state=542) # seed for repoducibility
X_train = train_df.drop(columns="Grades")
Y_train = train_df["Grades"]
X_test = test_df.drop(columns="Grades")
Y_test = test_df["Grades"]
# first build the formula
predictors = stu_performance.columns.drop("Grades")
formula = "Grades ~ " + " + ".join(predictors)
# fitting and predicting
full_model = smf.ols(formula=formula, data=train_df).fit()
Y_pred = full_model.predict(X_test)
test_mse = mean_squared_error(Y_test, Y_pred)
print(test_mse)
19.596503898487
full_model.summary()
| Dep. Variable: | Grades | R-squared: | 0.197 |
|---|---|---|---|
| Model: | OLS | Adj. R-squared: | 0.163 |
| Method: | Least Squares | F-statistic: | 5.708 |
| Date: | Fri, 21 Aug 2026 | Prob (F-statistic): | 2.27e-09 |
| Time: | 12:15:59 | Log-Likelihood: | -892.81 |
| No. Observations: | 316 | AIC: | 1814. |
| Df Residuals: | 302 | BIC: | 1866. |
| Df Model: | 13 | ||
| Covariance Type: | nonrobust |
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| Intercept | 14.9099 | 3.759 | 3.966 | 0.000 | 7.513 | 22.307 |
| age | -0.2938 | 0.200 | -1.466 | 0.144 | -0.688 | 0.101 |
| Medu | 0.6554 | 0.287 | 2.286 | 0.023 | 0.091 | 1.219 |
| Fedu | -0.1495 | 0.277 | -0.539 | 0.590 | -0.695 | 0.396 |
| traveltime | -0.2962 | 0.338 | -0.876 | 0.382 | -0.962 | 0.369 |
| studytime | 0.2276 | 0.300 | 0.758 | 0.449 | -0.363 | 0.818 |
| failures | -1.9248 | 0.338 | -5.701 | 0.000 | -2.589 | -1.260 |
| famrel | 0.1149 | 0.265 | 0.433 | 0.665 | -0.407 | 0.637 |
| freetime | 0.4377 | 0.258 | 1.699 | 0.090 | -0.069 | 0.945 |
| goout | -0.6981 | 0.255 | -2.733 | 0.007 | -1.201 | -0.196 |
| Dalc | -0.1399 | 0.359 | -0.389 | 0.697 | -0.847 | 0.567 |
| Walc | 0.5134 | 0.266 | 1.927 | 0.055 | -0.011 | 1.038 |
| health | -0.2987 | 0.170 | -1.752 | 0.081 | -0.634 | 0.037 |
| absences | 0.0240 | 0.029 | 0.827 | 0.409 | -0.033 | 0.081 |
| Omnibus: | 25.184 | Durbin-Watson: | 2.164 |
|---|---|---|---|
| Prob(Omnibus): | 0.000 | Jarque-Bera (JB): | 28.864 |
| Skew: | -0.696 | Prob(JB): | 5.40e-07 |
| Kurtosis: | 3.503 | Cond. No. | 320. |
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
Before proceeding, to better match glmnet in R which perform standardization, we scale our matrices.
from sklearn.preprocessing import StandardScaler
# Fit scaler on training data
scaler_X = StandardScaler()
X_train_scaled = scaler_X.fit_transform(X_train)
# Apply same scaler to test data
X_test_scaled = scaler_X.transform(X_test)
# Center Y
Y_mean = Y_train.mean()
Y_train_centered = Y_train - Y_mean
Y_test_centered = Y_test - Y_mean # for computing residuals/predictions
Ridge Regression¶
import numpy as np
from sklearn.linear_model import Ridge
alphas = np.logspace(-3, 3, 100)
coefs = []
intercepts = []
for a in alphas:
ridge = Ridge(alpha=a, fit_intercept=False) # intercept false to do it internally
ridge.fit(X_train, Y_train)
beta = ridge.coef_
coefs.append(beta)
# compute intercept "a0" like glmnet
mean_X = X_train.mean(axis=0)
mean_Y = Y_train.mean()
a0 = mean_Y - np.dot(mean_X, beta)
intercepts.append(a0)
beta_mat = np.column_stack(coefs)
a0_vec = np.array(intercepts)
ridge_lm = {
"lambda": np.array(alphas),
"beta": beta_mat,
"a0": a0_vec
}
len(ridge_lm["lambda"])
ridge_lm["beta"]
len(ridge_lm["a0"])
coef_mat = np.vstack([ridge_lm["a0"], ridge_lm["beta"]])
coef_mat.shape
# Check equivalence (should be ~0)
diff_sum = np.sum((coef_mat - np.vstack([ridge_lm["a0"], ridge_lm["beta"]]))**2)
print(diff_sum)
0.0
lambdas = ridge_lm["lambda"]
# y-axis: coefficients
betas = ridge_lm["beta"] # shape: (n_features, n_alphas)
plt.figure(figsize=(8,6))
# Plot each feature's coefficient across lambdas
for i in range(betas.shape[0]):
plt.plot(lambdas, betas[i, :], marker='o')
plt.xscale('log')
plt.xlabel('Lambda')
plt.ylabel('Coefficient')
plt.title('Ridge Coefficients vs Lambda')
plt.grid(True)
plt.tight_layout()
plt.show()
print(betas)
[[ 0.40065424 0.40065423 0.40065422 ... 0.44213502 0.44483945 0.4473361 ] [ 0.90278352 0.90278309 0.90278259 ... 0.38252752 0.35854272 0.33530576] [-0.0334491 -0.03344854 -0.03344789 ... 0.2253532 0.21806843 0.20998266] ... [ 0.63150802 0.63150725 0.63150635 ... 0.0860461 0.07879499 0.07282743] [-0.16322124 -0.16322116 -0.16322108 ... -0.01152455 -0.00295593 0.00536963] [ 0.00889445 0.00889445 0.00889445 ... 0.01289868 0.01370775 0.01462151]]
Computing intercepts
k = 1 # since python is 0-indexed
# manual intercept
mean_ridge = X_train.mean(axis=0)
intercept_manual = Y_train.mean() - np.sum(mean_ridge * beta_mat[:, k])
# stored intercept
intercept_model = a0_vec[k]
# both approaches give the same
diff_sq_sum = np.sum(
(Y_train.mean() - mean_ridge @ beta_mat - a0_vec)**2
)
print(diff_sq_sum)
1.735493991486226e-28
By computing the Residual sum of squares for the null model and at various lambda values, we can plot the fitted coefficients against the percent deviance. This is the percent of deviance explained on the training data, and is a measure of complexity of the model. We can see that the coefficients do not change much after a point the but the coefficients start blowing up a bit.
mean_Y = Y_train.mean()
RSS0 = np.sum((Y_train - mean_Y)**2)
RSS_lambda = []
for k in range(beta_mat.shape[1]):
y_pred = X_train @ beta_mat[:, k] + a0_vec[k]
rss = np.sum((Y_train - y_pred)**2)
RSS_lambda.append(rss)
RSS_lambda = np.array(RSS_lambda)
# Fraction of deviance explained
dev_explained = 1 - RSS_lambda / RSS0
# Plot coefficients vs fraction of deviance explained
plt.figure(figsize=(8,6))
for i in range(beta_mat.shape[0]):
plt.plot(dev_explained, beta_mat[i, :], marker='o', label=f'Feature {i+1}')
plt.xlabel('Fraction of deviance explained')
plt.ylabel('Coefficient')
plt.title('Ridge Coefficients vs Fraction of Deviance Explained')
plt.grid(True)
plt.tight_layout()
plt.show()
Comparing Python & R Results for Ridge¶
In this section we will highlight that Ridge in sklearn uses a different loss function from the elastic net used by glmnet. Some code will be shown for this conversion.
- This is only for those comparing against peers with R code and you mostly will not need to be worried about this.
from sklearn.model_selection import cross_val_score
N = X_train_scaled.shape[0]
alphas = np.exp(np.linspace(-2,4,100))*(N/2)
cv_scores = []
cv_stds = []
for alpha in alphas:
ridge = Ridge(alpha=alpha)
scores = cross_val_score(ridge, X_train_scaled, Y_train_centered, cv=10, scoring='neg_mean_squared_error')
cv_scores.append(-scores.mean()) # mean CV MSE
cv_stds.append(scores.std()/np.sqrt(10)) # SE across folds
cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)
alpha_min = alphas[np.argmin(cv_mse)]
alpha_1se = alphas[np.where(cv_mse <= cv_mse.min() + cv_std[np.argmin(cv_mse)])[0][-1]] # largest lambda whose mse is within one std of min MSE
print(alpha_min)
print(alpha_1se)
print(alpha_min*(2/N)) #equivalent in glmnet (https://stats.stackexchange.com/questions/160096/what-are-the-differences-between-ridge-regression-using-rs-glmnet-and-pythons)
print(alpha_1se*(2/N))
76.34956282799241 1895.8897224818531 0.4832250811898254 11.999302041024388
alphas
array([ 21.38297475, 22.71898898, 24.13847775, 25.64665657,
27.24906682, 28.95159612, 30.76049992, 32.68242453,
34.72443153, 36.8940237 , 39.19917259, 41.64834782,
44.25054821, 47.0153348 , 49.95286603, 53.07393504,
56.3900093 , 59.91327281, 63.6566708 , 67.63395735,
71.85974586, 76.34956283, 81.11990481, 86.18829908,
91.57336805, 97.29489762, 103.37390996, 109.83274069,
116.69512096, 123.98626466, 131.73296106, 139.9636732 ,
148.70864254, 158. , 167.87188407, 178.3605662 ,
189.50458412, 201.34488339, 213.92496786, 227.29105951,
241.4922683 , 256.58077258, 272.61201081, 289.64488527,
307.74197849, 326.96978313, 347.39894637, 369.10452943,
392.16628337, 416.66894213, 442.70253384, 470.36271162,
499.75110503, 530.97569345, 564.15120286, 599.39952736,
636.850177 , 676.64075367, 718.91745668, 763.83561987,
811.56028244, 862.26679522, 916.14146507, 973.38223932,
1034.19943311, 1098.81650212, 1167.47086363, 1240.41476879,
1317.9162295 , 1400.26000308, 1487.7486386 , 1580.70358847,
1679.46638953, 1784.39991794, 1895.88972248, 2014.3454411 ,
2140.20230606, 2273.92274303, 2415.99807021, 2566.95030345,
2727.33407432, 2897.73866793, 3078.79018808, 3271.15385769,
3475.53646304, 3692.68895056, 3923.40918607, 4168.54488625,
4428.99673334, 4705.72168447, 4999.73648773, 5312.12141789,
5644.02424561, 5996.66445458, 6371.33772216, 6769.42067998,
7192.37597204, 7641.7576287 , 8119.21677661, 8626.50770524])
cv_mse
array([18.22137631, 18.21558808, 18.20973165, 18.20382677, 18.19789572,
18.19196342, 18.18605756, 18.1802087 , 18.17445034, 18.16881905,
18.16335448, 18.15809941, 18.15309976, 18.14840457, 18.14406595,
18.140139 , 18.13668168, 18.13375464, 18.13142108, 18.12974641,
18.12879806, 18.1286451 , 18.12935788, 18.1310076 , 18.13366588,
18.13740428, 18.14229372, 18.14840397, 18.15580305, 18.16455664,
18.17472747, 18.18637472, 18.19955338, 18.21431368, 18.23070054,
18.24875298, 18.26850364, 18.28997833, 18.31319558, 18.33816634,
18.36489361, 18.3933723 , 18.42358898, 18.45552184, 18.48914063,
18.52440675, 18.56127329, 18.59968532, 18.63958002, 18.6808871 ,
18.72352914, 18.76742205, 18.81247557, 18.85859382, 18.90567594,
18.95361676, 19.00230747, 19.05163637, 19.10148966, 19.15175219,
19.20230827, 19.25304249, 19.30384044, 19.35458955, 19.40517978,
19.45550433, 19.50546031, 19.55494931, 19.60387791, 19.65215823,
19.69970821, 19.74645204, 19.7923203 , 19.83725021, 19.88118567,
19.92407729, 19.96588239, 20.00656483, 20.04609488, 20.08444903,
20.12160969, 20.15756493, 20.19230816, 20.22583777, 20.25815679,
20.2892725 , 20.3191961 , 20.34794227, 20.37552886, 20.40197651,
20.42730827, 20.45154933, 20.47472662, 20.4968686 , 20.5180049 ,
20.5381661 , 20.55738347, 20.57568878, 20.59311406, 20.60969143])
plt.figure(figsize=(8,6))
plt.errorbar(alphas, cv_mse, yerr=cv_std, fmt='-o', ecolor='lightgray', capsize=3)
plt.xscale('log')
plt.xlabel('Lambda')
plt.ylabel('CV MSE')
plt.title('Ridge CV')
plt.axvline(alpha_min, color='blue', linestyle='--', label='lambda.min')
plt.legend()
plt.grid(True)
plt.show()
We can also zoom in on a particular subspace of lambda values.
cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)
#change name to make compatible with R
lambda_seq = alphas
lambda_min_idx = np.argmin(cv_mse)
lambda_min = lambda_seq[lambda_min_idx]
lambda_min_idx = np.argmin(cv_mse)
lambda_min = lambda_seq[lambda_min_idx] # get lambda of minimum mse
plt.figure(figsize=(10, 6))
plt.errorbar(np.log(lambda_seq), cv_mse, yerr=cv_std, capsize=3)
plt.xlabel('log(λ)')
plt.ylabel('Mean-Squared Error')
plt.title('Ridge Regression Cross-Validation')
plt.axvline(np.log(lambda_min), color='red', linestyle='--', label=f'λ_min = {lambda_min:.4f}')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()
Prediction¶
Now we use these minimum values to perform prediction on the test data.
# Fit Ridge for all alphas
ridge_preds = {}
for alpha in alphas:
ridge = Ridge(alpha=alpha, fit_intercept=False)
ridge.fit(X_train_scaled, Y_train_centered)
ridge_preds[alpha] = ridge
# Using alpha_min
ridge_min = ridge_preds[alpha_min]
Ytest_ridge_min_pred = ridge_min.predict(X_test_scaled) + Y_train.mean() # add back mean if Y was centered
mse_min = np.mean((Ytest_ridge_min_pred - Y_test) ** 2)
print("MSE (alpha_min):", mse_min)
# Using alpha_1se
ridge_1se = ridge_preds[alpha_1se]
Ytest_ridge_1se_pred = ridge_1se.predict(X_test_scaled) + Y_train.mean()
mse_1se = np.mean((Ytest_ridge_1se_pred - Y_test) ** 2)
print("MSE (alpha_1se):", mse_1se)
MSE (alpha_min): 19.042536531458424 MSE (alpha_1se): 20.432799172444795
LASSO Regression¶
For Lasso we will use Lasso and LassoCV
from sklearn.linear_model import Lasso
n_samples = X_train_scaled.shape[0]
alphas = np.logspace(-4, 1, 100)
lasso_lm = Lasso()
# Plot coefficient paths (equivalent to plot(lasso.lm))
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
# Plot 1: Coefficients vs L1 norm
l1_norms = []
coeffs_path = []
for alpha in alphas:
lasso_temp = Lasso(alpha=alpha, max_iter=10000)
lasso_temp.fit(X_train_scaled, Y_train_centered)
coeffs_path.append(lasso_temp.coef_)
l1_norms.append(np.sum(np.abs(lasso_temp.coef_)))
coeffs_path = np.array(coeffs_path)
for i in range(coeffs_path.shape[1]):
ax1.plot(l1_norms, coeffs_path[:, i])
ax1.set_xlabel('||β||₁')
ax1.set_ylabel('Coefficients')
ax1.set_title('LASSO Coefficient Path')
for i in range(coeffs_path.shape[1]):
ax2.plot(np.log(alphas), coeffs_path[:, i])
ax2.set_xlabel('log(α)')
ax2.set_ylabel('Coefficients')
ax2.set_title('LASSO Coefficient Path')
plt.tight_layout()
plt.show()
# Extract degrees of freedom (number of non-zero coefficients)
df = [np.sum(coef != 0) for coef in coeffs_path]
print("Degrees of freedom:", df)
Degrees of freedom: [np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(12), np.int64(12), np.int64(12), np.int64(12), np.int64(11), np.int64(11), np.int64(10), np.int64(10), np.int64(10), np.int64(9), np.int64(9), np.int64(9), np.int64(9), np.int64(9), np.int64(8), np.int64(6), np.int64(5), np.int64(5), np.int64(4), np.int64(4), np.int64(4), np.int64(2), np.int64(2), np.int64(2), np.int64(2), np.int64(2), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0)]
# cross validation using LassoCV
lasso_cv = LassoCV(alphas=alphas, cv=10, max_iter=10000)
lasso_cv.fit(X_train_scaled, Y_train_centered)
# Cross-validation scores for plotting
cv_scores = []
cv_stds = []
for alpha in alphas:
lasso = Lasso(alpha=alpha, max_iter=10000)
scores = cross_val_score(lasso, X_train_scaled, Y_train_centered, cv=10, scoring='neg_mean_squared_error')
cv_scores.append(-scores.mean())
cv_stds.append(scores.std() / np.sqrt(len(scores)))
cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)
# plot(lasso.cv)
plt.figure(figsize=(10, 6))
plt.errorbar(np.log(alphas), cv_mse, yerr=cv_std, capsize=3)
plt.xlabel('log(α)')
plt.ylabel('Mean-Squared Error')
plt.title('LASSO Cross-Validation')
plt.axvline(np.log(lasso_cv.alpha_), color='red', linestyle='--', label=f'α_min = {lasso_cv.alpha_:.6f}')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()
# lambda.min and lambda.1se
alpha_min = lasso_cv.alpha_
alpha_min_idx = np.argmin(cv_mse)
alpha_1se_idx = np.where(cv_mse <= cv_mse[alpha_min_idx] + cv_std[alpha_min_idx])[0][-1]
alpha_1se = alphas[alpha_1se_idx]
print(f"alpha.min: {alpha_min}")
print(f"alpha.1se: {alpha_1se}")
alpha.min: 0.09545484566618337 alpha.1se: 1.232846739442066
# Extract coefficients
lasso_min = Lasso(alpha=alpha_min, max_iter=10000)
lasso_min.fit(X_train_scaled, Y_train_centered)
lasso_min_coef = lasso_min.coef_
lasso_1se = Lasso(alpha=alpha_1se, max_iter=10000)
lasso_1se.fit(X_train_scaled, Y_train_centered)
lasso_1se_coef = lasso_1se.coef_
# Display coefficients side by side
coef_comparison = np.column_stack([lasso_min_coef, lasso_1se_coef])
print("Coefficients (min, 1se):")
print(coef_comparison)
# Number of variables selected
print(f"Number of non-zero coefficients (1se): {np.sum(lasso_1se_coef != 0)}")
print(f"Number of non-zero coefficients (min): {np.sum(lasso_min_coef != 0)}")
Coefficients (min, 1se): [[-0.27135531 -0. ] [ 0.55895989 0. ] [-0. 0. ] [-0.12746314 -0. ] [ 0.0394935 0. ] [-1.37489405 -0.43215856] [ 0. 0. ] [ 0.26149608 0. ] [-0.52568281 -0. ] [-0. -0. ] [ 0.32234414 -0. ] [-0.29201279 -0. ] [ 0.10107942 0. ]] Number of non-zero coefficients (1se): 1 Number of non-zero coefficients (min): 10
# Names of selected variables (assuming you have feature names)
selected_features_1se = np.where(lasso_1se_coef != 0)[0]
selected_features_min = np.where(lasso_min_coef != 0)[0]
print(f"Selected features (1se): {selected_features_1se}")
print(f"Selected features (min): {selected_features_min}")
Selected features (1se): [5] Selected features (min): [ 0 1 3 4 5 7 8 10 11 12]
stu_performance.columns[selected_features_1se].tolist()
['failures']
stu_performance.columns[selected_features_min].tolist()
['age', 'Medu', 'traveltime', 'studytime', 'failures', 'freetime', 'goout', 'Walc', 'health', 'absences']
Predictions on test data¶
Y_test_lasso_min_pred = lasso_min.predict(X_test_scaled)
mse_min = np.mean((Y_test_lasso_min_pred - Y_test_centered)**2)
print(f"Test MSE (alpha.min): {mse_min}")
Y_test_lasso_1se_pred = lasso_1se.predict(X_test_scaled)
mse_1se = np.mean((Y_test_lasso_1se_pred - Y_test_centered)**2)
print(f"Test MSE (alpha.1se): {mse_1se}")
Test MSE (alpha.min): 19.2654246696838 Test MSE (alpha.1se): 20.58268401793688
Refit to Reduce Bias¶
We refit an ordinary linear regression model using the variables selected by `lambda.1se` and then use it for prediction.
from sklearn.linear_model import LinearRegression
# Get selected variables from 1se model
selected_indices_1se = np.where(lasso_1se_coef != 0)[0]
# Create subset of scaled data with only selected variables
tmp_X_train = X_train_scaled[:, selected_indices_1se]
tmp_X_test = X_test_scaled[:, selected_indices_1se]
# Refit linear regression on selected variables
lasso_refit = LinearRegression()
lasso_refit.fit(tmp_X_train, Y_train_centered)
# Predict on test data
Y_test_pred = lasso_refit.predict(tmp_X_test)
mse_refit = np.mean((Y_test_pred - Y_test_centered)**2)
print(f"Refitted model MSE: {mse_refit}")
Refitted model MSE: 19.227935377709333