Section 3.4 Python Code and Analysis¶

The Student Performance Example¶

In this final section we illustrate the practical implementation of the variable-selection and shrinkage methods developed earlier, using a real data example.

The analysis is based on the student-mat.csv data set, which contains information on student achievement in secondary education at two Portuguese schools. The original study is described in:

Cortez, P., & Silva, A. M. G. (2008). Using data mining to predict secondary school student performance. Proceedings of the 5th Annual Future Business Technology Conference.

The data are publicly available from the UCI Machine Learning Repository:
Student Performance data set.

The variables in the data set are the following:

  • school – student’s school (binary: GP – Gabriel Pereira or MS – Mousinho da Silveira)
  • sex – student’s sex (binary: F – female or M – male)
  • age – student’s age (numeric: from 15 to 22)
  • address – home address type (binary: U – urban or R – rural)
  • famsize – family size (binary: LE3 – ≤ 3 or GT3 – > 3)
  • Pstatus – parent’s cohabitation status (binary: T – living together or A – apart)
  • Medu – mother’s education (numeric: 0 – none, 1 – primary (4th grade), 2 – 5th–9th grade, 3 – secondary, 4 – higher education)
  • Fedu – father’s education (numeric: same coding as Medu)
  • Mjob – mother’s occupation (nominal: teacher, health, services, at_home, other)
  • Fjob – father’s occupation (nominal: same categories as Mjob)
  • reason – reason for choosing the school (nominal: home, reputation, course, other)
  • guardian – student’s guardian (nominal: mother, father, other)
  • traveltime – home-to-school travel time (numeric: 1 – <15 min, 2 – 15–30 min, 3 – 30–60 min, 4 – >1 hour)
  • studytime – weekly study time (numeric: 1 – <2 h, 2 – 2–5 h, 3 – 5–10 h, 4 – >10 h)
  • failures – number of past class failures (numeric: n if 1 ≤ n < 3, else 4)
  • schoolsup – extra educational support (binary: yes/no)
  • famsup – family educational support (binary: yes/no)
  • paid – extra paid classes in the course subject (binary: yes/no)
  • activities – extra-curricular activities (binary: yes/no)
  • nursery – attended nursery school (binary: yes/no)
  • higher – wants to take higher education (binary: yes/no)
  • internet – Internet access at home (binary: yes/no)
  • romantic – in a romantic relationship (binary: yes/no)
  • famrel – quality of family relationships (numeric: 1 – very bad to 5 – excellent)
  • freetime – free time after school (numeric: 1 – very low to 5 – very high)
  • goout – going out with friends (numeric: 1 – very low to 5 – very high)
  • Dalc – workday alcohol consumption (numeric: 1 – very low to 5 – very high)
  • Walc – weekend alcohol consumption (numeric: 1 – very low to 5 – very high)
  • health – current health status (numeric: 1 – very bad to 5 – very good)
  • absences – number of school absences (numeric: 0 to 93)
  • G1 – first-period grade (numeric: 0 to 20)
  • G2 – second-period grade (numeric: 0 to 20)
  • G3 – final grade (numeric: 0 to 20; the response variable)
In [112]:
import pandas as pd

stu_performance0 = pd.read_csv("student-mat.csv", sep=';')

Our goal is to predict the final grade (G3) using the penalized regression methods studied in this chapter. Because standard ridge and lasso do not handle categorical predictors directly (a group-lasso formulation would be required), we restrict attention to the numeric predictors only.

In [113]:
cols_to_drop = [
    'school', 'sex', 'address', 'famsize', 'Pstatus', 
    'Mjob', 'Fjob', 'reason', 'guardian', 'schoolsup', 
    'famsup', 'paid', 'activities', 'nursery', 'higher', 
    'internet', 'romantic', 'G1', 'G2'
]
stu_performance = stu_performance0.drop(columns=cols_to_drop) # data frame without categorical variables

# manually renaming response to 'Grades'
stu_performance = stu_performance.rename(columns={'G3': 'Grades'})

n = stu_performance.shape[0]  # sample size (number of rows)
p = stu_performance.shape[1] - 1  # number of non-intercept predictors (columns - 1)

stu_performance.head()
Out[113]:
age Medu Fedu traveltime studytime failures famrel freetime goout Dalc Walc health absences Grades
0 18 4 4 2 2 0 4 3 4 1 1 3 6 6
1 17 1 1 1 2 0 5 3 3 1 1 3 4 6
2 15 1 1 1 2 3 4 3 2 2 3 3 10 10
3 15 4 2 1 3 0 3 2 2 1 1 5 2 15
4 16 3 3 1 2 0 4 3 2 1 2 5 4 10

We begin by fitting the full linear model that includes every numeric predictor:

In [114]:
import statsmodels.formula.api as smf

# due to error with patsy ~. formula, workaround by building the formula "explicitly"
predictors = stu_performance.columns.drop("Grades")
formula = "Grades ~ " + " + ".join(predictors)

performance_full = smf.ols(formula=formula, data=stu_performance).fit()
performance_full.summary()
Out[114]:
OLS Regression Results
Dep. Variable: Grades R-squared: 0.183
Model: OLS Adj. R-squared: 0.155
Method: Least Squares F-statistic: 6.564
Date: Fri, 21 Aug 2026 Prob (F-statistic): 2.53e-11
Time: 12:15:49 Log-Likelihood: -1121.3
No. Observations: 395 AIC: 2271.
Df Residuals: 381 BIC: 2326.
Df Model: 13
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 13.5370 3.337 4.056 0.000 6.975 20.099
age -0.2562 0.179 -1.433 0.153 -0.608 0.095
Medu 0.5849 0.257 2.280 0.023 0.081 1.089
Fedu -0.0913 0.255 -0.359 0.720 -0.592 0.409
traveltime -0.4018 0.314 -1.278 0.202 -1.020 0.216
studytime 0.2658 0.269 0.988 0.324 -0.263 0.795
failures -1.8315 0.312 -5.870 0.000 -2.445 -1.218
famrel 0.2802 0.245 1.142 0.254 -0.202 0.763
freetime 0.3414 0.231 1.478 0.140 -0.113 0.796
goout -0.6264 0.222 -2.824 0.005 -1.062 -0.190
Dalc -0.1372 0.320 -0.428 0.669 -0.767 0.493
Walc 0.3531 0.238 1.482 0.139 -0.116 0.822
health -0.1743 0.156 -1.116 0.265 -0.481 0.133
absences 0.0320 0.028 1.158 0.247 -0.022 0.086
Omnibus: 28.620 Durbin-Watson: 2.062
Prob(Omnibus): 0.000 Jarque-Bera (JB): 32.727
Skew: -0.677 Prob(JB): 7.82e-08
Kurtosis: 3.391 Cond. No. 314.


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

 

A simple (but inefficient) approach to model selection is to compare the full model with a reduced model that retains only the apparently significant variables, using a partial F-test:

 

In [115]:
import statsmodels.api as sm
from statsmodels.stats.anova import anova_lm

# first we fit a reduced model
formula_reduced = 'Grades ~ failures + goout + Medu'
performance_red = smf.ols(formula=formula_reduced, data=stu_performance).fit()

anova_results = anova_lm(performance_red, performance_full)
print(anova_results)
   df_resid          ssr  df_diff     ss_diff         F    Pr(>F)
0     391.0  6956.920171      0.0         NaN       NaN       NaN
1     381.0  6756.704779     10.0  200.215392  1.128983  0.338977

In this case, we fail to reject the null so the reduced model is preferred. However, this approach is not efficient nor optimal. Therefore, we use algorithms to search the space of models and a penalty to penalize the inclusion of terms that are not as useful.

Subset Selection via Level-wise algorithms¶


In [116]:
import itertools
import statsmodels.api as sm
import pandas as pd
import numpy as np

# Data
X = stu_performance.drop(columns="Grades")
y = stu_performance["Grades"]

# Full model MSE for Mallows' Cp
sigma2_full = performance_full.mse_resid

# max subset size
nvmax = X.shape[1]  

# we will store the best model per subset size
best_per_size = []

for k in range(1, nvmax + 1):
    best_model = None
    best_rsq = -np.inf
    
    # Loop over all combinations of size k using itertools combinations
    for combo in itertools.combinations(X.columns, k):
        X_subset = sm.add_constant(X[list(combo)])
        model = sm.OLS(y, X_subset).fit()
        
        if model.rsquared > best_rsq:
            best_rsq = model.rsquared
            best_model = {
                "k": k,
                "predictors": combo,
                "rsq": model.rsquared,
                "adj_rsq": model.rsquared_adj,
                "bic": model.bic,
                "aic": model.aic,
                "rss": model.ssr
            }
    
    # Compute Mallows' Cp
    best_model["cp"] = best_model["rss"] / sigma2_full - (len(y) - 2*len(best_model["predictors"]))
    best_per_size.append(best_model)


rs = pd.DataFrame(best_per_size)

The `rs` object above contains all the different criteria for all the models that have been evaluated.

Below we extract the $R^2$, Adjusted $R^2$, $AIC$, $BIC$ and $C_p$-Mallows criteria for the 13 models that have been evaluated:


In [117]:
rs
Out[117]:
k predictors rsq adj_rsq bic aic rss cp
0 1 (failures,) 0.129899 0.127685 2279.346318 2271.388546 7195.656554 12.751803
1 2 (Medu, failures) 0.148313 0.143968 2276.875838 2264.939180 7043.370273 6.164618
2 3 (Medu, failures, goout) 0.158767 0.152313 2277.976507 2262.060964 6956.920171 3.289832
3 4 (Medu, failures, freetime, goout) 0.163370 0.154789 2281.788229 2261.893800 6918.855608 3.143431
4 5 (Medu, traveltime, failures, freetime, goout) 0.166832 0.156123 2286.128964 2262.255649 6890.221033 3.528772
5 6 (Medu, traveltime, failures, freetime, goout, ... 0.169663 0.156823 2290.763350 2262.911150 6866.807998 4.208548
6 7 (age, Medu, traveltime, failures, freetime, go... 0.172408 0.157439 2295.434493 2263.603407 6844.111358 4.928720
7 8 (age, Medu, traveltime, failures, freetime, go... 0.174691 0.157586 2300.322023 2264.512051 6825.227682 5.863899
8 9 (age, Medu, traveltime, failures, famrel, free... 0.177358 0.158127 2305.022777 2265.233919 6803.178460 6.620578
9 10 (age, Medu, traveltime, failures, famrel, free... 0.180038 0.158685 2309.712285 2265.944542 6781.007415 7.370388
10 11 (age, Medu, traveltime, studytime, failures, f... 0.182315 0.158831 2314.592990 2266.846361 6762.181008 8.308796
11 12 (age, Medu, traveltime, studytime, failures, f... 0.182701 0.157027 2320.385250 2268.659736 6758.986840 10.128682
12 13 (age, Medu, Fedu, traveltime, studytime, failu... 0.182977 0.155100 2326.230748 2270.526348 6756.704779 12.000000

Going by $r^2$, the best model is the 13th as expected. By adjusted $r^2$ it is the 11th; by CP it is the 4th, and by BIC the best model is the 2nd one.

We can identify which variables are included by looking at the predictors column

In [118]:
print(rs["predictors"].iloc[1])
('Medu', 'failures')

For example, the model with two variables contains only failures, Medu and the intercept. The model with only one feature contains onlyfailures, and the intercept.

We can also compute AIC and BIC "by hand" for the sequence of models:

In [119]:
rss = rs['rss']
msize = np.arange(1, p + 1)
Aic = n * np.log(rss / n) + 2 * msize
Bic = n * np.log(rss / n) + msize * np.log(n)
In [120]:
Aic
Out[120]:
0     1148.427105
1     1141.977739
2     1139.099523
3     1138.932359
4     1139.294208
5     1139.949709
6     1140.641966
7     1141.550610
8     1142.272478
9     1142.983101
10    1143.884920
11    1145.698294
12    1147.564906
Name: rss, dtype: float64

AIC chooses the 4th model.

In [121]:
Bic
Out[121]:
0     1152.405991
1     1149.935511
2     1151.036180
3     1154.847902
4     1159.188637
5     1163.823023
6     1168.494166
7     1173.381696
8     1178.082450
9     1182.771958
10    1187.652663
11    1193.444923
12    1199.290421
Name: rss, dtype: float64

As we saw before, BIC selects the 2nd model.
We can also plot the results for the various models. So, we have

In [122]:
import matplotlib.pyplot as plt


fig, axes = plt.subplots(2, 2)


axes[0, 0].plot(msize, rs['adj_rsq'])
axes[0, 0].set_xlabel("No. of Parameters")
axes[0, 0].set_ylabel("Adjusted Rsquare")


axes[0, 1].plot(msize, rs['cp'])
axes[0, 1].set_xlabel("No. of Parameters")
axes[0, 1].set_ylabel("Mallow's Cp")

axes[1, 0].plot(msize, Aic)
axes[1, 0].set_xlabel("No. of Parameters")
axes[1, 0].set_ylabel("AIC")

# R: plot(msize, Bic, xlab="No. of Parameters", ylab = "BIC")
axes[1, 1].plot(msize, Bic)
axes[1, 1].set_xlabel("No. of Parameters")
axes[1, 1].set_ylabel("BIC")

plt.tight_layout() # cleans it up
plt.show()
No description has been provided for this image

As is typical, AIC selects a larger model than BIC. Although the BIC-optimal model does not attain the absolute minimum AIC, its AIC value is very close to the optimum.

In [123]:
best_k_aic = Aic.idxmin()
best_k_bic = Bic.idxmin()

comparison_df = pd.DataFrame({
    f'AIC-Best ({best_k_aic} features)': rs.loc[best_k_aic],
    f'BIC-Best ({best_k_bic} features)': rs.loc[best_k_bic]
})

comparison_df
Out[123]:
AIC-Best (3 features) BIC-Best (1 features)
k 4 2
predictors (Medu, failures, freetime, goout) (Medu, failures)
rsq 0.16337 0.148313
adj_rsq 0.154789 0.143968
bic 2281.788229 2276.875838
aic 2261.8938 2264.93918
rss 6918.855608 7043.370273
cp 3.143431 6.164618

Note that the manually computed AIC and BIC values are different from those calculated using the sm package. But the difference is only a constant and so the same models are selected.

What are the 2nd and 3rd best models in terms of AIC/BIC? This cannot be answered by looking at the AIC/BIC plots shown above. Instead, we need to run the following code.

In [124]:
all_models = []
for k in range(1, p + 1):
    for combo in itertools.combinations(X.columns, k):
        feature_set = list(combo)
        model = sm.OLS(y, sm.add_constant(X[feature_set])).fit()
        all_models.append({
            "num_features": len(feature_set),
            "features": feature_set,
            "rss": model.ssr
        })
all_models_df = pd.DataFrame(all_models)

# Get the top 3 models for each subset size, based on lowest RSS
nbest = 3
top_nbest_id = all_models_df.groupby('num_features')['rss'].nsmallest(nbest).index.get_level_values(1)
rs_1_df = all_models_df.loc[top_nbest_id]

# Replicate msize.1
msize_1 = rs_1_df['num_features']
In [125]:
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

Aic_1 = n * np.log(rs_1_df['rss'] / n) + 2 * msize_1
Bic_1 = n * np.log(rs_1_df['rss'] / n) + msize_1 * np.log(n)

axes[0].scatter(msize_1, Aic_1)
axes[0].set_xlabel("No. of Parameters")
axes[0].set_ylabel("AIC")

axes[1].scatter(msize_1, Bic_1)
axes[1].set_xlabel("No. of Parameters")
axes[1].set_ylabel("BIC")
plt.show()
No description has been provided for this image
In [126]:
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

axes[0].scatter(msize_1, Aic_1)
axes[0].set_xlabel("No. of Parameters")
axes[0].set_ylabel("AIC")
axes[0].set_ylim(1135, 1152)

axes[1].scatter(msize_1, Bic_1)
axes[1].set_xlabel("No. of Parameters")
axes[1].set_ylabel("BIC")
axes[1].set_ylim(1140, 1230)
plt.show()
No description has been provided for this image
In [127]:
which_df_1 = pd.DataFrame(False, index=rs_1_df.index, columns=X.columns)
for index, row in rs_1_df.iterrows():
    which_df_1.loc[index, row['features']] = True

# Add AIC/BIC to the df to sort
which_df_1['Aic'] = Aic_1
which_df_1['Bic'] = Bic_1

print("--- Top 3 models by AIC ---")
print(which_df_1.sort_values('Aic').head(3).drop(columns=['Aic', 'Bic']))

print("\n--- Top 3 models by BIC ---")
print(which_df_1.sort_values('Bic').head(3).drop(columns=['Aic', 'Bic']))
--- Top 3 models by AIC ---
        age  Medu   Fedu  traveltime  studytime  failures  famrel  freetime  \
712   False  True  False       False      False      True   False      True   
186   False  True  False       False      False      True   False     False   
1741  False  True  False        True      False      True   False      True   

      goout   Dalc   Walc  health  absences  
712    True  False  False   False     False  
186    True  False  False   False     False  
1741   True  False  False   False     False  

--- Top 3 models by BIC ---
       age   Medu   Fedu  traveltime  studytime  failures  famrel  freetime  \
28   False   True  False       False      False      True   False     False   
186  False   True  False       False      False      True   False     False   
5    False  False  False       False      False      True   False     False   

     goout   Dalc   Walc  health  absences  
28   False  False  False   False     False  
186   True  False  False   False     False  
5    False  False  False   False     False  

 

Subset Selection via the step fucntion¶

 

We can use the stepwise approach to search possible models moving forward, backwards or in both directions (stepwise).

Python does not have a prebuilt function that mimicks the step function R. However, we can mimick this code using a for loop. Note that in the following function the default value for $k =2$, and uses AIC. The criteria can be changed to BIC by changing $k$ appropriately.


In [128]:
import numpy as np
import statsmodels.formula.api as smf

def stepwise_selection(model, k=2): # For BIC, use k=np.log(len(model.model.data.frame))
    """
    Performs bidirectional (forward/backward) variable selection based on a criterion.
    Takes a fitted statsmodels model as input, mimicking R's step().
    """
    df = model.model.data.frame
    response = model.model.endog_names
    
    # Get initial predictors from the model, excluding the intercept
    initial_predictors = {p for p in model.model.exog_names if p != 'Intercept'}
    
    # Get all possible predictors from the dataframe
    all_predictors = {col for col in df.columns if col != response}
    
    included = initial_predictors
    excluded = all_predictors - included
    
    while True:
        # Define the current best model and its criterion
        current_formula = f"{response} ~ {' + '.join(included) or '1'}"
        current_model = smf.ols(current_formula, data=df).fit()
        # The fix is here: using .df_model instead of .df_modelwc
        current_criterion = -2 * current_model.llf + k * current_model.df_model
        
        best_candidate_criterion = current_criterion
        best_move = None

        # Check backward moves (dropping a variable)
        for var in included:
            formula = f"{response} ~ {' + '.join(included - {var}) or '1'}"
            temp_model = smf.ols(formula, data=df).fit()
            criterion = -2 * temp_model.llf + k * temp_model.df_model
            if criterion < best_candidate_criterion:
                best_candidate_criterion = criterion
                best_move = ('drop', var)

        # Check forward moves (adding a variable)
        for var in excluded:
            formula = f"{response} ~ {' + '.join(included | {var}) or '1'}"
            temp_model = smf.ols(formula, data=df).fit()
            criterion = -2 * temp_model.llf + k * temp_model.df_model
            if criterion < best_candidate_criterion:
                best_candidate_criterion = criterion
                best_move = ('add', var)
        
        # If a better model was found, make the move
        if best_move:
            move_type, var = best_move
            if move_type == 'drop':
                included.remove(var)
                excluded.add(var)
            else: # 'add'
                excluded.remove(var)
                included.add(var)
        else:
            # If no move improves the criterion, stop
            break
            
    final_formula = f"{response} ~ {' + '.join(included) or '1'}"
    return smf.ols(final_formula, data=df).fit()
In [129]:
stepAIC = stepwise_selection(performance_full, k=2)
In [130]:
n = len(stu_performance)
stepBIC = stepwise_selection(performance_full, k=np.log(n))
In [131]:
stepAIC.summary()
Out[131]:
OLS Regression Results
Dep. Variable: Grades R-squared: 0.163
Model: OLS Adj. R-squared: 0.155
Method: Least Squares F-statistic: 19.04
Date: Fri, 21 Aug 2026 Prob (F-statistic): 2.57e-14
Time: 12:15:59 Log-Likelihood: -1125.9
No. Observations: 395 AIC: 2262.
Df Residuals: 390 BIC: 2282.
Df Model: 4
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 9.8868 0.970 10.194 0.000 7.980 11.794
goout -0.5064 0.201 -2.524 0.012 -0.901 -0.112
freetime 0.3254 0.222 1.465 0.144 -0.111 0.762
Medu 0.6190 0.201 3.087 0.002 0.225 1.013
failures -1.9505 0.297 -6.558 0.000 -2.535 -1.366
Omnibus: 37.170 Durbin-Watson: 2.041
Prob(Omnibus): 0.000 Jarque-Bera (JB): 45.218
Skew: -0.769 Prob(JB): 1.52e-10
Kurtosis: 3.618 Cond. No. 25.6


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.


In [132]:
stepBIC.summary()
Out[132]:
OLS Regression Results
Dep. Variable: Grades R-squared: 0.148
Model: OLS Adj. R-squared: 0.144
Method: Least Squares F-statistic: 34.13
Date: Fri, 21 Aug 2026 Prob (F-statistic): 2.16e-14
Time: 12:15:59 Log-Likelihood: -1129.5
No. Observations: 395 AIC: 2265.
Df Residuals: 392 BIC: 2277.
Df Model: 2
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 9.4821 0.621 15.267 0.000 8.261 10.703
Medu 0.5845 0.201 2.911 0.004 0.190 0.979
failures -2.0168 0.296 -6.824 0.000 -2.598 -1.436
Omnibus: 35.775 Durbin-Watson: 2.001
Prob(Omnibus): 0.000 Jarque-Bera (JB): 42.955
Skew: -0.760 Prob(JB): 4.70e-10
Kurtosis: 3.545 Cond. No. 9.63


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



Notice that all of the variables in stepBIC are strictly included in the variables of stepAIC
In [133]:
stepAIC.model.exog_names
Out[133]:
['Intercept', 'goout', 'freetime', 'Medu', 'failures']
In [134]:
stepBIC.model.exog_names
Out[134]:
['Intercept', 'Medu', 'failures']
In [135]:
set(stepBIC.model.exog_names).issubset(set(stepAIC.model.exog_names))
Out[135]:
True

LASSO and RIDGE Regression¶

SciKitLearn has packages for Lasso and Ridge

In [136]:
from sklearn.linear_model import LassoCV, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
In [137]:
X = stu_performance.drop(columns="Grades")
y = stu_performance["Grades"]

We'll first break up the data into train (80%) and test (20%) and create a full model.

In [138]:
#split the data and assign
train_df, test_df = train_test_split(stu_performance, test_size=0.2, random_state=542) # seed for repoducibility
X_train = train_df.drop(columns="Grades")
Y_train = train_df["Grades"]
X_test = test_df.drop(columns="Grades")
Y_test = test_df["Grades"]
# first build the formula
predictors = stu_performance.columns.drop("Grades")
formula = "Grades ~ " + " + ".join(predictors)  
# fitting and predicting
full_model = smf.ols(formula=formula, data=train_df).fit()
Y_pred = full_model.predict(X_test)

test_mse = mean_squared_error(Y_test, Y_pred)
print(test_mse)
19.596503898487
In [139]:
full_model.summary()
Out[139]:
OLS Regression Results
Dep. Variable: Grades R-squared: 0.197
Model: OLS Adj. R-squared: 0.163
Method: Least Squares F-statistic: 5.708
Date: Fri, 21 Aug 2026 Prob (F-statistic): 2.27e-09
Time: 12:15:59 Log-Likelihood: -892.81
No. Observations: 316 AIC: 1814.
Df Residuals: 302 BIC: 1866.
Df Model: 13
Covariance Type: nonrobust
coef std err t P>|t| [0.025 0.975]
Intercept 14.9099 3.759 3.966 0.000 7.513 22.307
age -0.2938 0.200 -1.466 0.144 -0.688 0.101
Medu 0.6554 0.287 2.286 0.023 0.091 1.219
Fedu -0.1495 0.277 -0.539 0.590 -0.695 0.396
traveltime -0.2962 0.338 -0.876 0.382 -0.962 0.369
studytime 0.2276 0.300 0.758 0.449 -0.363 0.818
failures -1.9248 0.338 -5.701 0.000 -2.589 -1.260
famrel 0.1149 0.265 0.433 0.665 -0.407 0.637
freetime 0.4377 0.258 1.699 0.090 -0.069 0.945
goout -0.6981 0.255 -2.733 0.007 -1.201 -0.196
Dalc -0.1399 0.359 -0.389 0.697 -0.847 0.567
Walc 0.5134 0.266 1.927 0.055 -0.011 1.038
health -0.2987 0.170 -1.752 0.081 -0.634 0.037
absences 0.0240 0.029 0.827 0.409 -0.033 0.081
Omnibus: 25.184 Durbin-Watson: 2.164
Prob(Omnibus): 0.000 Jarque-Bera (JB): 28.864
Skew: -0.696 Prob(JB): 5.40e-07
Kurtosis: 3.503 Cond. No. 320.


Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



Before proceeding, to better match glmnet in R which perform standardization, we scale our matrices.
In [140]:
from sklearn.preprocessing import StandardScaler

# Fit scaler on training data
scaler_X = StandardScaler()
X_train_scaled = scaler_X.fit_transform(X_train)

# Apply same scaler to test data
X_test_scaled = scaler_X.transform(X_test)

# Center Y
Y_mean = Y_train.mean()
Y_train_centered = Y_train - Y_mean
Y_test_centered = Y_test - Y_mean  # for computing residuals/predictions

Ridge Regression¶

In [141]:
import numpy as np
from sklearn.linear_model import Ridge

alphas = np.logspace(-3, 3, 100)

coefs = []       
intercepts = []  

for a in alphas:
    ridge = Ridge(alpha=a, fit_intercept=False)  # intercept false to do it internally
    ridge.fit(X_train, Y_train)

    beta = ridge.coef_
    coefs.append(beta)

    # compute intercept "a0" like glmnet
    mean_X = X_train.mean(axis=0)
    mean_Y = Y_train.mean()
    a0 = mean_Y - np.dot(mean_X, beta)
    intercepts.append(a0)

beta_mat = np.column_stack(coefs)   
a0_vec = np.array(intercepts) 

ridge_lm = {
    "lambda": np.array(alphas),
    "beta": beta_mat,
    "a0": a0_vec
}
In [142]:
len(ridge_lm["lambda"])

ridge_lm["beta"]

len(ridge_lm["a0"])

coef_mat = np.vstack([ridge_lm["a0"], ridge_lm["beta"]])
coef_mat.shape

# Check equivalence (should be ~0)
diff_sum = np.sum((coef_mat - np.vstack([ridge_lm["a0"], ridge_lm["beta"]]))**2)
print(diff_sum)
0.0
In [143]:
lambdas = ridge_lm["lambda"]

# y-axis: coefficients
betas = ridge_lm["beta"]  # shape: (n_features, n_alphas)

plt.figure(figsize=(8,6))

# Plot each feature's coefficient across lambdas
for i in range(betas.shape[0]):
    plt.plot(lambdas, betas[i, :], marker='o')

plt.xscale('log')  
plt.xlabel('Lambda')
plt.ylabel('Coefficient')
plt.title('Ridge Coefficients vs Lambda')
plt.grid(True)
plt.tight_layout()
plt.show()
No description has been provided for this image
In [144]:
print(betas)
[[ 0.40065424  0.40065423  0.40065422 ...  0.44213502  0.44483945
   0.4473361 ]
 [ 0.90278352  0.90278309  0.90278259 ...  0.38252752  0.35854272
   0.33530576]
 [-0.0334491  -0.03344854 -0.03344789 ...  0.2253532   0.21806843
   0.20998266]
 ...
 [ 0.63150802  0.63150725  0.63150635 ...  0.0860461   0.07879499
   0.07282743]
 [-0.16322124 -0.16322116 -0.16322108 ... -0.01152455 -0.00295593
   0.00536963]
 [ 0.00889445  0.00889445  0.00889445 ...  0.01289868  0.01370775
   0.01462151]]

Computing intercepts

In [145]:
k = 1  # since python is 0-indexed

# manual intercept
mean_ridge = X_train.mean(axis=0)
intercept_manual = Y_train.mean() - np.sum(mean_ridge * beta_mat[:, k])

# stored intercept
intercept_model = a0_vec[k]

# both approaches give the same
diff_sq_sum = np.sum(
    (Y_train.mean() - mean_ridge @ beta_mat - a0_vec)**2
)

print(diff_sq_sum)
1.735493991486226e-28

By computing the Residual sum of squares for the null model and at various lambda values, we can plot the fitted coefficients against the percent deviance. This is the percent of deviance explained on the training data, and is a measure of complexity of the model. We can see that the coefficients do not change much after a point the but the coefficients start blowing up a bit.

In [146]:
mean_Y = Y_train.mean()
RSS0 = np.sum((Y_train - mean_Y)**2)

RSS_lambda = []
for k in range(beta_mat.shape[1]):
    y_pred = X_train @ beta_mat[:, k] + a0_vec[k]
    rss = np.sum((Y_train - y_pred)**2)
    RSS_lambda.append(rss)

RSS_lambda = np.array(RSS_lambda)

# Fraction of deviance explained
dev_explained = 1 - RSS_lambda / RSS0

# Plot coefficients vs fraction of deviance explained
plt.figure(figsize=(8,6))
for i in range(beta_mat.shape[0]):
    plt.plot(dev_explained, beta_mat[i, :], marker='o', label=f'Feature {i+1}')

plt.xlabel('Fraction of deviance explained')
plt.ylabel('Coefficient')
plt.title('Ridge Coefficients vs Fraction of Deviance Explained')
plt.grid(True)
plt.tight_layout()
plt.show()
No description has been provided for this image

Comparing Python & R Results for Ridge¶

In this section we will highlight that Ridge in sklearn uses a different loss function from the elastic net used by glmnet. Some code will be shown for this conversion.

  • This is only for those comparing against peers with R code and you mostly will not need to be worried about this.
In [147]:
from sklearn.model_selection import cross_val_score
N = X_train_scaled.shape[0]
alphas = np.exp(np.linspace(-2,4,100))*(N/2)
cv_scores = []
cv_stds = []
for alpha in alphas:
    ridge = Ridge(alpha=alpha)
    scores = cross_val_score(ridge, X_train_scaled, Y_train_centered, cv=10, scoring='neg_mean_squared_error')
    cv_scores.append(-scores.mean())        # mean CV MSE
    cv_stds.append(scores.std()/np.sqrt(10))  # SE across folds
    

cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)

alpha_min = alphas[np.argmin(cv_mse)]
alpha_1se = alphas[np.where(cv_mse <= cv_mse.min() + cv_std[np.argmin(cv_mse)])[0][-1]] # largest lambda whose mse is within one std of min MSE

print(alpha_min)
print(alpha_1se)

print(alpha_min*(2/N)) #equivalent in glmnet (https://stats.stackexchange.com/questions/160096/what-are-the-differences-between-ridge-regression-using-rs-glmnet-and-pythons)
print(alpha_1se*(2/N))
76.34956282799241
1895.8897224818531
0.4832250811898254
11.999302041024388
In [148]:
alphas
Out[148]:
array([  21.38297475,   22.71898898,   24.13847775,   25.64665657,
         27.24906682,   28.95159612,   30.76049992,   32.68242453,
         34.72443153,   36.8940237 ,   39.19917259,   41.64834782,
         44.25054821,   47.0153348 ,   49.95286603,   53.07393504,
         56.3900093 ,   59.91327281,   63.6566708 ,   67.63395735,
         71.85974586,   76.34956283,   81.11990481,   86.18829908,
         91.57336805,   97.29489762,  103.37390996,  109.83274069,
        116.69512096,  123.98626466,  131.73296106,  139.9636732 ,
        148.70864254,  158.        ,  167.87188407,  178.3605662 ,
        189.50458412,  201.34488339,  213.92496786,  227.29105951,
        241.4922683 ,  256.58077258,  272.61201081,  289.64488527,
        307.74197849,  326.96978313,  347.39894637,  369.10452943,
        392.16628337,  416.66894213,  442.70253384,  470.36271162,
        499.75110503,  530.97569345,  564.15120286,  599.39952736,
        636.850177  ,  676.64075367,  718.91745668,  763.83561987,
        811.56028244,  862.26679522,  916.14146507,  973.38223932,
       1034.19943311, 1098.81650212, 1167.47086363, 1240.41476879,
       1317.9162295 , 1400.26000308, 1487.7486386 , 1580.70358847,
       1679.46638953, 1784.39991794, 1895.88972248, 2014.3454411 ,
       2140.20230606, 2273.92274303, 2415.99807021, 2566.95030345,
       2727.33407432, 2897.73866793, 3078.79018808, 3271.15385769,
       3475.53646304, 3692.68895056, 3923.40918607, 4168.54488625,
       4428.99673334, 4705.72168447, 4999.73648773, 5312.12141789,
       5644.02424561, 5996.66445458, 6371.33772216, 6769.42067998,
       7192.37597204, 7641.7576287 , 8119.21677661, 8626.50770524])
In [149]:
cv_mse
Out[149]:
array([18.22137631, 18.21558808, 18.20973165, 18.20382677, 18.19789572,
       18.19196342, 18.18605756, 18.1802087 , 18.17445034, 18.16881905,
       18.16335448, 18.15809941, 18.15309976, 18.14840457, 18.14406595,
       18.140139  , 18.13668168, 18.13375464, 18.13142108, 18.12974641,
       18.12879806, 18.1286451 , 18.12935788, 18.1310076 , 18.13366588,
       18.13740428, 18.14229372, 18.14840397, 18.15580305, 18.16455664,
       18.17472747, 18.18637472, 18.19955338, 18.21431368, 18.23070054,
       18.24875298, 18.26850364, 18.28997833, 18.31319558, 18.33816634,
       18.36489361, 18.3933723 , 18.42358898, 18.45552184, 18.48914063,
       18.52440675, 18.56127329, 18.59968532, 18.63958002, 18.6808871 ,
       18.72352914, 18.76742205, 18.81247557, 18.85859382, 18.90567594,
       18.95361676, 19.00230747, 19.05163637, 19.10148966, 19.15175219,
       19.20230827, 19.25304249, 19.30384044, 19.35458955, 19.40517978,
       19.45550433, 19.50546031, 19.55494931, 19.60387791, 19.65215823,
       19.69970821, 19.74645204, 19.7923203 , 19.83725021, 19.88118567,
       19.92407729, 19.96588239, 20.00656483, 20.04609488, 20.08444903,
       20.12160969, 20.15756493, 20.19230816, 20.22583777, 20.25815679,
       20.2892725 , 20.3191961 , 20.34794227, 20.37552886, 20.40197651,
       20.42730827, 20.45154933, 20.47472662, 20.4968686 , 20.5180049 ,
       20.5381661 , 20.55738347, 20.57568878, 20.59311406, 20.60969143])
In [150]:
plt.figure(figsize=(8,6))
plt.errorbar(alphas, cv_mse, yerr=cv_std, fmt='-o', ecolor='lightgray', capsize=3)
plt.xscale('log')
plt.xlabel('Lambda')
plt.ylabel('CV MSE')
plt.title('Ridge CV')
plt.axvline(alpha_min, color='blue', linestyle='--', label='lambda.min')
plt.legend()
plt.grid(True)
plt.show()
No description has been provided for this image

We can also zoom in on a particular subspace of lambda values.

In [151]:
cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)

#change name to make compatible with R

lambda_seq = alphas
lambda_min_idx = np.argmin(cv_mse)
lambda_min = lambda_seq[lambda_min_idx]

lambda_min_idx = np.argmin(cv_mse)
lambda_min = lambda_seq[lambda_min_idx] # get lambda of minimum mse

plt.figure(figsize=(10, 6))
plt.errorbar(np.log(lambda_seq), cv_mse, yerr=cv_std, capsize=3)
plt.xlabel('log(λ)')
plt.ylabel('Mean-Squared Error')
plt.title('Ridge Regression Cross-Validation')
plt.axvline(np.log(lambda_min), color='red', linestyle='--', label=f'λ_min = {lambda_min:.4f}')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()
No description has been provided for this image

Prediction¶

Now we use these minimum values to perform prediction on the test data.

In [152]:
# Fit Ridge for all alphas
ridge_preds = {}
for alpha in alphas:
    ridge = Ridge(alpha=alpha, fit_intercept=False)
    ridge.fit(X_train_scaled, Y_train_centered)
    ridge_preds[alpha] = ridge

# Using alpha_min
ridge_min = ridge_preds[alpha_min]
Ytest_ridge_min_pred = ridge_min.predict(X_test_scaled) + Y_train.mean()  # add back mean if Y was centered
mse_min = np.mean((Ytest_ridge_min_pred - Y_test) ** 2)
print("MSE (alpha_min):", mse_min)

# Using alpha_1se
ridge_1se = ridge_preds[alpha_1se]
Ytest_ridge_1se_pred = ridge_1se.predict(X_test_scaled) + Y_train.mean()
mse_1se = np.mean((Ytest_ridge_1se_pred - Y_test) ** 2)
print("MSE (alpha_1se):", mse_1se)
MSE (alpha_min): 19.042536531458424
MSE (alpha_1se): 20.432799172444795

LASSO Regression¶

For Lasso we will use Lasso and LassoCV

In [153]:
from sklearn.linear_model import Lasso
In [154]:
n_samples = X_train_scaled.shape[0]
alphas = np.logspace(-4, 1, 100)
lasso_lm = Lasso()

# Plot coefficient paths (equivalent to plot(lasso.lm))
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))

# Plot 1: Coefficients vs L1 norm
l1_norms = []
coeffs_path = []
for alpha in alphas:
    lasso_temp = Lasso(alpha=alpha, max_iter=10000)
    lasso_temp.fit(X_train_scaled, Y_train_centered)
    coeffs_path.append(lasso_temp.coef_)
    l1_norms.append(np.sum(np.abs(lasso_temp.coef_)))
    
coeffs_path = np.array(coeffs_path)
for i in range(coeffs_path.shape[1]):
    ax1.plot(l1_norms, coeffs_path[:, i])
ax1.set_xlabel('||β||₁')
ax1.set_ylabel('Coefficients')
ax1.set_title('LASSO Coefficient Path')


for i in range(coeffs_path.shape[1]):
    ax2.plot(np.log(alphas), coeffs_path[:, i])
ax2.set_xlabel('log(α)')
ax2.set_ylabel('Coefficients')
ax2.set_title('LASSO Coefficient Path')
plt.tight_layout()
plt.show()
No description has been provided for this image
In [155]:
# Extract degrees of freedom (number of non-zero coefficients)
df = [np.sum(coef != 0) for coef in coeffs_path]
print("Degrees of freedom:", df)
Degrees of freedom: [np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(13), np.int64(12), np.int64(12), np.int64(12), np.int64(12), np.int64(11), np.int64(11), np.int64(10), np.int64(10), np.int64(10), np.int64(9), np.int64(9), np.int64(9), np.int64(9), np.int64(9), np.int64(8), np.int64(6), np.int64(5), np.int64(5), np.int64(4), np.int64(4), np.int64(4), np.int64(2), np.int64(2), np.int64(2), np.int64(2), np.int64(2), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(1), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0), np.int64(0)]
In [156]:
# cross validation using LassoCV
lasso_cv = LassoCV(alphas=alphas, cv=10, max_iter=10000)
lasso_cv.fit(X_train_scaled, Y_train_centered)

# Cross-validation scores for plotting
cv_scores = []
cv_stds = []
for alpha in alphas:
    lasso = Lasso(alpha=alpha, max_iter=10000)
    scores = cross_val_score(lasso, X_train_scaled, Y_train_centered, cv=10, scoring='neg_mean_squared_error')
    cv_scores.append(-scores.mean())
    cv_stds.append(scores.std() / np.sqrt(len(scores)))

cv_mse = np.array(cv_scores)
cv_std = np.array(cv_stds)
In [157]:
# plot(lasso.cv)
plt.figure(figsize=(10, 6))
plt.errorbar(np.log(alphas), cv_mse, yerr=cv_std, capsize=3)
plt.xlabel('log(α)')
plt.ylabel('Mean-Squared Error')
plt.title('LASSO Cross-Validation')
plt.axvline(np.log(lasso_cv.alpha_), color='red', linestyle='--', label=f'α_min = {lasso_cv.alpha_:.6f}')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

# lambda.min and lambda.1se
alpha_min = lasso_cv.alpha_
alpha_min_idx = np.argmin(cv_mse)
alpha_1se_idx = np.where(cv_mse <= cv_mse[alpha_min_idx] + cv_std[alpha_min_idx])[0][-1]
alpha_1se = alphas[alpha_1se_idx]

print(f"alpha.min: {alpha_min}")
print(f"alpha.1se: {alpha_1se}")
No description has been provided for this image
alpha.min: 0.09545484566618337
alpha.1se: 1.232846739442066
In [158]:
# Extract coefficients
lasso_min = Lasso(alpha=alpha_min, max_iter=10000)
lasso_min.fit(X_train_scaled, Y_train_centered)
lasso_min_coef = lasso_min.coef_

lasso_1se = Lasso(alpha=alpha_1se, max_iter=10000)
lasso_1se.fit(X_train_scaled, Y_train_centered)
lasso_1se_coef = lasso_1se.coef_

# Display coefficients side by side
coef_comparison = np.column_stack([lasso_min_coef, lasso_1se_coef])
print("Coefficients (min, 1se):")
print(coef_comparison)

# Number of variables selected
print(f"Number of non-zero coefficients (1se): {np.sum(lasso_1se_coef != 0)}")
print(f"Number of non-zero coefficients (min): {np.sum(lasso_min_coef != 0)}")
Coefficients (min, 1se):
[[-0.27135531 -0.        ]
 [ 0.55895989  0.        ]
 [-0.          0.        ]
 [-0.12746314 -0.        ]
 [ 0.0394935   0.        ]
 [-1.37489405 -0.43215856]
 [ 0.          0.        ]
 [ 0.26149608  0.        ]
 [-0.52568281 -0.        ]
 [-0.         -0.        ]
 [ 0.32234414 -0.        ]
 [-0.29201279 -0.        ]
 [ 0.10107942  0.        ]]
Number of non-zero coefficients (1se): 1
Number of non-zero coefficients (min): 10
In [159]:
# Names of selected variables (assuming you have feature names)
selected_features_1se = np.where(lasso_1se_coef != 0)[0]
selected_features_min = np.where(lasso_min_coef != 0)[0]
print(f"Selected features (1se): {selected_features_1se}")
print(f"Selected features (min): {selected_features_min}")
Selected features (1se): [5]
Selected features (min): [ 0  1  3  4  5  7  8 10 11 12]
In [160]:
stu_performance.columns[selected_features_1se].tolist()
Out[160]:
['failures']
In [161]:
stu_performance.columns[selected_features_min].tolist()
Out[161]:
['age',
 'Medu',
 'traveltime',
 'studytime',
 'failures',
 'freetime',
 'goout',
 'Walc',
 'health',
 'absences']

Predictions on test data¶


In [162]:
Y_test_lasso_min_pred = lasso_min.predict(X_test_scaled)
mse_min = np.mean((Y_test_lasso_min_pred - Y_test_centered)**2)
print(f"Test MSE (alpha.min): {mse_min}")

Y_test_lasso_1se_pred = lasso_1se.predict(X_test_scaled)
mse_1se = np.mean((Y_test_lasso_1se_pred - Y_test_centered)**2)
print(f"Test MSE (alpha.1se): {mse_1se}")
Test MSE (alpha.min): 19.2654246696838
Test MSE (alpha.1se): 20.58268401793688

Refit to Reduce Bias¶


We refit an ordinary linear regression model using the variables selected by `lambda.1se` and then use it for prediction.
In [163]:
from sklearn.linear_model import LinearRegression

# Get selected variables from 1se model
selected_indices_1se = np.where(lasso_1se_coef != 0)[0]

# Create subset of scaled data with only selected variables
tmp_X_train = X_train_scaled[:, selected_indices_1se]
tmp_X_test = X_test_scaled[:, selected_indices_1se]

# Refit linear regression on selected variables
lasso_refit = LinearRegression()
lasso_refit.fit(tmp_X_train, Y_train_centered)

# Predict on test data
Y_test_pred = lasso_refit.predict(tmp_X_test)
mse_refit = np.mean((Y_test_pred - Y_test_centered)**2)
print(f"Refitted model MSE: {mse_refit}")
Refitted model MSE: 19.227935377709333