The Student Performance Example

 

In this final section we illustrate the practical implementation of the variable-selection and shrinkage methods developed earlier, using a real data example.

The analysis is based on the student-mat.csv data set, which contains information on student achievement in secondary education at two Portuguese schools. The original study is described in:

Cortez, P., & Silva, A. M. G. (2008). Using data mining to predict secondary school student performance. Proceedings of the 5th Annual Future Business Technology Conference.

The data are publicly available from the UCI Machine Learning Repository:
Student Performance data set.

The variables in the data set are the following:

stu_performance0 <- read.csv2("student-mat.csv", header = TRUE)

Our goal is to predict the final grade (G3) using the penalized regression methods studied in this chapter. Because standard ridge and lasso do not handle categorical predictors directly (a group-lasso formulation would be required), we restrict attention to the numeric predictors only.

## Keep only numeric predictors
stu_performance <- stu_performance0[, -c(1, 2, 4, 5, 6, 9, 10, 11, 12, 16, 17, 18, 19, 20, 21, 22, 23, 31, 32)]

## Rename the response
names(stu_performance)[14] <- "Grades"
head(stu_performance)
##   age Medu Fedu traveltime studytime failures famrel freetime goout Dalc Walc
## 1  18    4    4          2         2        0      4        3     4    1    1
## 2  17    1    1          1         2        0      5        3     3    1    1
## 3  15    1    1          1         2        3      4        3     2    2    3
## 4  15    4    2          1         3        0      3        2     2    1    1
## 5  16    3    3          1         2        0      4        3     2    1    2
## 6  16    4    3          1         2        0      5        4     2    1    2
##   health absences Grades
## 1      3        6      6
## 2      3        4      6
## 3      3       10     10
## 4      5        2     15
## 5      5        4     10
## 6      5       10     15
n <- nrow(stu_performance)          # sample size
p <- ncol(stu_performance) - 1      # number of predictors (excluding intercept)

We begin by fitting the full linear model that includes every numeric predictor:

performance.full = lm(Grades ~ ., data=stu_performance)
summary(performance.full)
## 
## Call:
## lm(formula = Grades ~ ., data = stu_performance)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -12.2769  -2.1753   0.4053   2.7927   8.7122 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 13.53702    3.33724   4.056 6.05e-05 ***
## age         -0.25623    0.17880  -1.433  0.15268    
## Medu         0.58494    0.25650   2.280  0.02313 *  
## Fedu        -0.09135    0.25465  -0.359  0.72000    
## traveltime  -0.40184    0.31439  -1.278  0.20197    
## studytime    0.26583    0.26903   0.988  0.32373    
## failures    -1.83146    0.31201  -5.870 9.49e-09 ***
## famrel       0.28018    0.24537   1.142  0.25422    
## freetime     0.34139    0.23096   1.478  0.14021    
## goout       -0.62638    0.22179  -2.824  0.00499 ** 
## Dalc        -0.13723    0.32034  -0.428  0.66862    
## Walc         0.35312    0.23834   1.482  0.13928    
## health      -0.17429    0.15620  -1.116  0.26521    
## absences     0.03197    0.02760   1.158  0.24745    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.211 on 381 degrees of freedom
## Multiple R-squared:  0.183,  Adjusted R-squared:  0.1551 
## F-statistic: 6.564 on 13 and 381 DF,  p-value: 2.525e-11

 

A simple (but inefficient) approach to model selection is to compare the full model with a reduced model that retains only the apparently significant variables, using a partial F-test:

 

performance.red = lm(Grades ~ failures + goout + Medu, data=stu_performance)

## Model comparison based on a partial F test
anova(performance.red, performance.full)
## Analysis of Variance Table
## 
## Model 1: Grades ~ failures + goout + Medu
## Model 2: Grades ~ age + Medu + Fedu + traveltime + studytime + failures + 
##     famrel + freetime + goout + Dalc + Walc + health + absences
##   Res.Df    RSS Df Sum of Sq     F Pr(>F)
## 1    391 6956.9                          
## 2    381 6756.7 10    200.22 1.129  0.339

 

In this particular case we fail to reject the null hypothesis, so the reduced model is preferred by the F-test. However, this ad-hoc strategy is neither systematic nor optimal. We therefore turn to formal search algorithms that explore the model space while applying an explicit complexity penalty.

 

Subset Selection via Level-wise algorithms & the leaps package

 

library(leaps)
regsubsets_model <- regsubsets(Grades ~ ., data = stu_performance, nvmax = p)
rs <- summary(regsubsets_model)

 

The object rs stores the information criteria for every model examined. By default regsubsets considers models of size up to 8; we have overridden this limit with nvmax = p so that all model sizes from 1 to 13 are evaluated.

Below we extract the \(R^2\), Adjusted \(R^2\), \(AIC\), \(BIC\) and \(C_p\)-Mallows criteria for the 13 models that have been evaluated:

 

## Extract the models' R^2
rs$rsq
##  [1] 0.1298989 0.1483134 0.1587670 0.1633698 0.1668323 0.1696634 0.1724079
##  [8] 0.1746913 0.1773575 0.1800384 0.1823149 0.1827012 0.1829771

The best model is the 13th model (as expected).  

## Extract the models' Adjusted R^2
rs$adjr2
##  [1] 0.1276849 0.1439681 0.1523125 0.1547890 0.1561232 0.1568231 0.1574385
##  [8] 0.1575865 0.1581269 0.1586853 0.1588305 0.1570269 0.1550997

The best model is the 11th model.  

## Extract the models' Cp Mallows
rs$cp
##  [1] 14.751803  8.164618  5.289832  5.143431  5.528772  6.208548  6.928720
##  [8]  7.863899  8.620578  9.370388 10.308796 12.128682 14.000000

The best model is the 4th model.  

## Extract the models' BIC
rs$bic
##  [1] -43.004859 -45.475340 -44.374670 -40.562949 -36.222213 -31.587827
##  [7] -26.916684 -22.029154 -17.328400 -12.638892  -7.758187  -1.965927
## [13]   3.879571

The best model is the 2nd model.

 

The matrix rs$which indicates which variables appear in each model (TRUE = included):

 

rs$which
##    (Intercept)   age  Medu  Fedu traveltime studytime failures famrel freetime
## 1         TRUE FALSE FALSE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 2         TRUE FALSE  TRUE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 3         TRUE FALSE  TRUE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 4         TRUE FALSE  TRUE FALSE      FALSE     FALSE     TRUE  FALSE     TRUE
## 5         TRUE FALSE  TRUE FALSE       TRUE     FALSE     TRUE  FALSE     TRUE
## 6         TRUE FALSE  TRUE FALSE       TRUE     FALSE     TRUE  FALSE     TRUE
## 7         TRUE  TRUE  TRUE FALSE       TRUE     FALSE     TRUE  FALSE     TRUE
## 8         TRUE  TRUE  TRUE FALSE       TRUE     FALSE     TRUE  FALSE     TRUE
## 9         TRUE  TRUE  TRUE FALSE       TRUE     FALSE     TRUE   TRUE     TRUE
## 10        TRUE  TRUE  TRUE FALSE       TRUE     FALSE     TRUE   TRUE     TRUE
## 11        TRUE  TRUE  TRUE FALSE       TRUE      TRUE     TRUE   TRUE     TRUE
## 12        TRUE  TRUE  TRUE FALSE       TRUE      TRUE     TRUE   TRUE     TRUE
## 13        TRUE  TRUE  TRUE  TRUE       TRUE      TRUE     TRUE   TRUE     TRUE
##    goout  Dalc  Walc health absences
## 1  FALSE FALSE FALSE  FALSE    FALSE
## 2  FALSE FALSE FALSE  FALSE    FALSE
## 3   TRUE FALSE FALSE  FALSE    FALSE
## 4   TRUE FALSE FALSE  FALSE    FALSE
## 5   TRUE FALSE FALSE  FALSE    FALSE
## 6   TRUE FALSE  TRUE  FALSE    FALSE
## 7   TRUE FALSE FALSE  FALSE     TRUE
## 8   TRUE FALSE  TRUE  FALSE     TRUE
## 9   TRUE FALSE  TRUE   TRUE    FALSE
## 10  TRUE FALSE  TRUE   TRUE     TRUE
## 11  TRUE FALSE  TRUE   TRUE     TRUE
## 12  TRUE  TRUE  TRUE   TRUE     TRUE
## 13  TRUE  TRUE  TRUE   TRUE     TRUE

 

For example, Model 1 only contains failures, and the intercept.

 

We can also compute AIC and BIC “by hand” for the sequence of models:

 

msize <- 1:p
par(mfrow = c(1, 2))
Aic <- n * log(rs$rss / n) + 2 * msize
Bic <- n * log(rs$rss / n) + msize * log(n)
Aic
##  [1] 1148.427 1141.978 1139.100 1138.932 1139.294 1139.950 1140.642 1141.551
##  [9] 1142.272 1142.983 1143.885 1145.698 1147.565

AIC chooses the 4th model.

Bic
##  [1] 1152.406 1149.936 1151.036 1154.848 1159.189 1163.823 1168.494 1173.382
##  [9] 1178.082 1182.772 1187.653 1193.445 1199.290

As we saw before, BIC selects the 2nd model.   We can also plot the results for the various models. So, we have  

par(mfrow = c(2, 2))
plot(msize, rs$adjr2, xlab = "Number of parameters", ylab = "Adjusted R²")
plot(msize, rs$cp,    xlab = "Number of parameters", ylab = "Mallows’ Cp")
plot(msize, Aic,      xlab = "Number of parameters", ylab = "AIC")
plot(msize, Bic,      xlab = "Number of parameters", ylab = "BIC")

 

As is typical, AIC selects a larger model than BIC. Although the BIC-optimal model does not attain the absolute minimum AIC, its AIC value is very close to the optimum.

 

cbind(rs$which[which.min(Aic),], rs$which[which.min(Bic), ])
##              [,1]  [,2]
## (Intercept)  TRUE  TRUE
## age         FALSE FALSE
## Medu         TRUE  TRUE
## Fedu        FALSE FALSE
## traveltime  FALSE FALSE
## studytime   FALSE FALSE
## failures     TRUE  TRUE
## famrel      FALSE FALSE
## freetime     TRUE FALSE
## goout        TRUE FALSE
## Dalc        FALSE FALSE
## Walc        FALSE FALSE
## health      FALSE FALSE
## absences    FALSE FALSE

 

(Note that the BIC reported by leaps differs from our hand calculation by a constant; the ranking of models is therefore identical.)

 

cbind(rs$bic, Bic, rs$bic - Bic)
##                       Bic          
##  [1,] -43.004859 1152.406 -1195.411
##  [2,] -45.475340 1149.936 -1195.411
##  [3,] -44.374670 1151.036 -1195.411
##  [4,] -40.562949 1154.848 -1195.411
##  [5,] -36.222213 1159.189 -1195.411
##  [6,] -31.587827 1163.823 -1195.411
##  [7,] -26.916684 1168.494 -1195.411
##  [8,] -22.029154 1173.382 -1195.411
##  [9,] -17.328400 1178.082 -1195.411
## [10,] -12.638892 1182.772 -1195.411
## [11,]  -7.758187 1187.653 -1195.411
## [12,]  -1.965927 1193.445 -1195.411
## [13,]   3.879571 1199.290 -1195.411

 

To examine the second- and third-best models we request the three best models of each size:

 

regsubsets_model.1 = regsubsets(Grades ~ ., data=stu_performance, nbest = 3, nvmax = p)
rs.1 = summary(regsubsets_model.1)
#rs.1$which
msize.1 = apply(rs.1$which, 1, sum) - 1
par(mfrow=c(1,2))
Aic.1 = n*log(rs.1$rss/n) + 2*msize.1
Bic.1 = n*log(rs.1$rss/n) + msize.1*log(n)
plot(msize.1, Aic.1, xlab="No. of Parameters", ylab = "AIC")
plot(msize.1, Bic.1, xlab="No. of Parameters", ylab = "BIC")

par(mfrow=c(1,2))
plot(msize.1, Aic.1, ylim = c(1135, 1152), xlab="No. of Parameters", ylab = "AIC")
plot(msize.1, Bic.1, ylim = c(1140, 1230), xlab="No. of Parameters", ylab = "BIC")

## Top three models by AIC
rs.1$which[order(Aic.1)[1:3], ]
##   (Intercept)   age Medu  Fedu traveltime studytime failures famrel freetime
## 4        TRUE FALSE TRUE FALSE      FALSE     FALSE     TRUE  FALSE     TRUE
## 3        TRUE FALSE TRUE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 5        TRUE FALSE TRUE FALSE       TRUE     FALSE     TRUE  FALSE     TRUE
##   goout  Dalc  Walc health absences
## 4  TRUE FALSE FALSE  FALSE    FALSE
## 3  TRUE FALSE FALSE  FALSE    FALSE
## 5  TRUE FALSE FALSE  FALSE    FALSE
## Top three models by BIC
rs.1$which[order(Bic.1)[1:3], ]
##   (Intercept)   age  Medu  Fedu traveltime studytime failures famrel freetime
## 2        TRUE FALSE  TRUE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 3        TRUE FALSE  TRUE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
## 1        TRUE FALSE FALSE FALSE      FALSE     FALSE     TRUE  FALSE    FALSE
##   goout  Dalc  Walc health absences
## 2 FALSE FALSE FALSE  FALSE    FALSE
## 3  TRUE FALSE FALSE  FALSE    FALSE
## 1 FALSE FALSE FALSE  FALSE    FALSE

 

Subset Selection via the step Fucntion

 

The step function implements forward, backward, and stepwise search. By default it uses AIC; setting k = log(n) yields BIC.

 

## Stepwise Selection with AIC

stepAIC = step(performance.full, trace=0, direction="both")  
#stepAIC = step(performance.full, direction="both")
## Stepwise Selection with BIC

n = dim(stu_performance)[1]
stepBIC = step(performance.full, trace=0, direction="both", k=log(n))      

 

The selected variables can be extracted as follows:

 

sel.var.AIC = attr(stepAIC$terms, "term.labels")
sel.var.BIC = attr(stepBIC$terms, "term.labels")
sel.var.AIC
## [1] "Medu"     "failures" "freetime" "goout"
length(sel.var.AIC)
## [1] 4
length(sel.var.BIC)
## [1] 2
c("Medu", "failures","freetime", "goout") %in% sel.var.AIC
## [1] TRUE TRUE TRUE TRUE
sel.var.BIC %in% sel.var.AIC
## [1] TRUE TRUE

 

LASSO and RIDGE Regression

 

The package glmnet implements both ridge (\(\alpha = 0\)) and lasso (\(\alpha = 1\)). Additional examples can be found at https://web.stanford.edu/~hastie/glmnet/glmnet_beta.html

 

library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 5.0
## The input in the glmnet function should be in matrix format:

X = as.matrix(stu_performance[, names(stu_performance) != "Grades"])
Y = stu_performance$Grades

 

We randomly split the data into an 80 % training set and a 20 % test set. Model selection and coefficient estimation are performed on the training data; predictive performance is evaluated on the held-out test set.

## Training/ Testing Data Sets
set.seed(598)                       # for reproducibility
ntest  <- round(n * 0.2)
ntrain <- n - ntest
test.id <- sample(1:n, ntest)
Ytest  <- Y[test.id]

 

For reference we also fit the ordinary least-squares model on the training data and compute its test-set MSE:

 

full.model <- lm(Y[-test.id] ~ ., data = stu_performance[-test.id, names(stu_performance) != "Grades"])
Ytest.pred <- predict(full.model, newdata = stu_performance[test.id, names(stu_performance) != "Grades"])
mean((Ytest - Ytest.pred)^2)
## [1] 22.31678

 

Ridge Regression

 

Fit a set of ridge regression models using glmnet with the default \(\lambda\) sequence. Selecting \(\alpha=0\) to run a ridge  

We plot the fitted output of the ridge regression. We can use the xvar argument to decide what is plotted on the x-axis. We can also label the curves with the variable index numbers simply by setting label = TRUE.

 

ridge.lm = glmnet(X[-test.id, ], Y[-test.id], alpha = 0)
plot(ridge.lm , label = TRUE, xvar = "lambda")

  We can obtain the \(\lambda\) values  

length(ridge.lm$lambda)
## [1] 100
## remove the length function to see the lambda values

  and the coefficients :  

dim(ridge.lm $beta)       ## coefficients for  non-intercept predictors
## [1]  13 100
length(ridge.lm $a0)      ## intercept
## [1] 100
## You can also retrieve coefficients (including intercept) using coef(myridge)
dim(coef(ridge.lm ))
## [1]  14 100
## The two coefficient matrices should be the same
sum((coef(ridge.lm ) - rbind(ridge.lm $a0, ridge.lm $beta))^2)
## [1] 0

  How are the intercepts computed?  

k = 2; 

## Compute the "intercept" by hand:
mean.ridge = apply(X[-test.id, ], 2, mean)  ## mean vector for training X
mean(Y[-test.id]) - sum(mean.ridge * ridge.lm $beta[, k])
## [1] 10.60974
## Extract intercept from the estimated coefficients
ridge.lm$a0[k]            ## intercept for lambda = myridge$lambda[k]
##       s1 
## 10.60974
## Both approaches give the same value
sum((mean(Y[-test.id]) - mean.ridge %*% ridge.lm $beta  - ridge.lm $a0)^2)
## [1] 1.072851e-27

 

We can also choose xvar="dev" to plot the fitted coefficients against the percent deviance. This is the percent of deviance explained on the training data, and is a measure of complexity of the model. We can see that the coefficients do not change much after a point the but the coefficients start blowing up a bit.

 

plot(ridge.lm, xvar = "dev", label = TRUE)

 

Cross-Validation & Path Plot

  If we want to use cross-validation to choose \(\lambda\), then we can use the function cv.glmnet. This function is using a \(K\)-fold cross validation method to obtain the “optimal lambda”. The cv.glmnet has the attribute nfolds to select the number of folds we prefer, the option foldid to specify our own folds, and type.measure to choose the loss function (deviance or mse for squared loss, and mae for mean absolute error).

 

ridge.cv = cv.glmnet(X[-test.id, ], Y[-test.id], alpha = 0) 
## the default in nfolds is 10

  Some basic output information can be obtained  

print(ridge.cv)
## 
## Call:  cv.glmnet(x = X[-test.id, ], y = Y[-test.id], alpha = 0) 
## 
## Measure: Mean-Squared Error 
## 
##     Lambda Index Measure    SE Nonzero
## min   1.50    75   17.37 1.639      13
## 1se  46.96    38   19.01 2.014      13

As a side note, the cv.glmnet function allows parallel computing for large scale problems.

The results are stored in

ridge.cv$cvm mean CV errors ridge.cv$cvsd: standard errors of CVm

  We can also plot the result of the cv.glmnet:

plot(ridge.cv)

 

The CV performance continues to drop when \(\lambda\) gets smaller. So, we provide R with a \(\lambda\) sequence dense in \([-2, 2]\):  

lambda.seq = exp(seq(-2, 4, length=100))
ridge.cv.seq = cv.glmnet(X[-test.id,], Y[-test.id], alpha=0, lambda=lambda.seq)  
plot(ridge.cv.seq)

We can still use the coef and predict methods for cv.glmnet objects in the same way as for the glmnet. In this case, we also have two special strings supported by \(s\) (the values of \(\lambda\) requested):

  • lambda.min: the value of lambda that achieves the minimum MSE (i.e. cvm – the left dashed vertical line)

  • lambda.1se: the largest value of lambda (i.e., the largest regularization, the smallest df) whose MSE (i.e. cvm) is within one-standard-error of the cvm of lambda.min – i.e. the smallest MSE (– the left dashed vertical line).

Note: by definition lambda.1se (right dashed vertical line) is always bigger than or equal to lambda.min (left dashed vertical line).

Check how lambda.min and lambda.1se are computed.

names(ridge.cv.seq)
##  [1] "lambda"     "cvm"        "cvsd"       "cvup"       "cvlo"      
##  [6] "nzero"      "call"       "name"       "glmnet.fit" "lambda.min"
## [11] "lambda.1se" "index"
## Find lambda.min
ridge.cv.seq$lambda.min
## [1] 1.833195
ridge.cv.seq$lambda[which.min(ridge.cv.seq$cvm)]
## [1] 1.833195
## Find lambda.1se
ridge.cv.seq$lambda.1se
## [1] 54.59815
tmp.id = which.min(ridge.cv.seq$cvm)
max(ridge.cv.seq$lambda[ridge.cv.seq$cvm < ridge.cv.seq$cvm[tmp.id] + ridge.cv.seq$cvsd[tmp.id]])
## [1] 54.59815

 

Prediction

 

Next we apply the two ridge regression models—one with lambda.min and one with lambda.1se on a new test data set and report the corresponding mean squared prediction errors:  

ridge.lambda.pred = glmnet(X[-test.id,], Y[-test.id], alpha=0, lambda = lambda.seq)
## Using the lambda.min
Ytest.ridge.min.pred = predict(ridge.lambda.pred, s = ridge.cv.seq$lambda.min, newx=X[test.id,])
mean((Ytest.ridge.min.pred - Y[test.id])^2)
## [1] 22.53208
## Using lambda.1se
Ytest.ridge.1se.pred = predict(ridge.lambda.pred, s = ridge.cv.seq$lambda.1se, newx=X[test.id,])
mean((Ytest.ridge.1se.pred - Y[test.id])^2)
## [1] 25.47849

 

LASSO Regression

  If we want to use LASSO instead of Ridge, we need to set \(\alpha=1\) in the glmnet function. In this case, we have  

lasso.lm = glmnet(X[-test.id,], Y[-test.id], alpha = 1)

  Using cross validation, we can see the CV path plot:  

par(mfrow = c(1, 2))
plot(lasso.lm, label=TRUE, xvar = "norm")   ## norm is the default value for xvar
plot(lasso.lm, label=TRUE, xvar = "lambda")

par(mfrow=c(1,1))

  We can extract the number of non-zero coefficients for each value of \(\lambda\).  

lasso.lm$df
##  [1]  0  1  1  1  1  2  2  2  2  2  3  3  4  4  4  4  4  4  5  6  6  6  6  7  7
## [26]  7  7  7  7  7  7  7  7  7  9  9  9  9  9  9  9  9  9 10 12 12 12 12 12 12
## [51] 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12

  As with the ridge regression example, we can still use cross-validation by calling the cv.glmnet:  

lasso.cv = cv.glmnet(X[-test.id, ], Y[-test.id], alpha = 1)
plot(lasso.cv)

 

From the cv.glmnet object we can still compute lambda.min and lambda.1se in the same way as before:

 

## Computing lambda.min
lasso.cv$lambda.min
## [1] 0.1190689
## Computing lambda.1se
lasso.cv$lambda.1se
## [1] 0.9219072

  We can also extract the LASSO coefficients:  

lasso.min.pred.coef = predict(lasso.lm, s=lasso.cv$lambda.min, type="coefficients");
lasso.1se.pred.coef = predict(lasso.lm, s=lasso.cv$lambda.1se, type="coefficients");
cbind(lasso.min.pred.coef, lasso.1se.pred.coef);
## 14 x 2 sparse Matrix of class "dgCMatrix"
##              s=0.1190689 s=0.9219072
## (Intercept) 14.920661406 10.75737536
## age         -0.281420530  .         
## Medu         0.600006298  0.02508545
## Fedu         .            .         
## traveltime   .            .         
## studytime    .            .         
## failures    -1.540566778 -0.76873824
## famrel       .            .         
## freetime     0.254866691  .         
## goout       -0.422725540  .         
## Dalc         .            .         
## Walc         .            .         
## health      -0.103790831  .         
## absences     0.008985155  .
## number of variables selected (including the intercept)
sum(lasso.1se.pred.coef != 0)
## [1] 3
sum(lasso.min.pred.coef != 0)
## [1] 8
## names of selected non-intercept variables
row.names(lasso.1se.pred.coef)[which(lasso.1se.pred.coef != 0)[-1]]
## [1] "Medu"     "failures"
row.names(lasso.min.pred.coef)[which(lasso.min.pred.coef != 0)[-1]]
## [1] "age"      "Medu"     "failures" "freetime" "goout"    "health"   "absences"
Prediction

Apply the two fitted models for prediction on test data:

lasso.lambda.pred = glmnet(X[-test.id, ], Y[-test.id], alpha = 1)
## Using the lambda.min
Ytest.lasso.min.pred = predict(lasso.lambda.pred, s = lasso.cv$lambda.min, newx = X[test.id,])
mean((Ytest.lasso.min.pred - Y[test.id])^2)
## [1] 22.301
## Using lambda.1se
Ytest.lasso.1se.pred = predict(lasso.lambda.pred, s = lasso.cv$lambda.1se, newx=X[test.id,])
mean((Ytest.lasso.1se.pred - Y[test.id])^2)
## [1] 23.71601

 

Debiasing via Refitting

A common post-processing step is to refit an ordinary least-squares model on the variables selected by the lasso (here using lambda.1se) and then use that unbiased estimator for prediction:

lasso.coef.1se = predict(lasso.lambda.pred, s = lasso.cv$lambda.1se, type="coefficients")
variables.lasso = row.names(lasso.coef.1se)[which(lasso.coef.1se != 0)[-1]]
variables.lasso
## [1] "Medu"     "failures"
tmp.X <- X[, colnames(X) %in% variables.lasso, drop = FALSE]

train.df <- data.frame(Y = Y[-test.id], tmp.X[-test.id, , drop = FALSE])
lasso.refit <- coef(lm(Y ~ ., data = train.df))

# prediction on the test set
Ytest.pred <- lasso.refit[1] + as.matrix(tmp.X[test.id, , drop = FALSE]) %*% lasso.refit[-1]
mean((Ytest.pred - Y[test.id])^2)
## [1] 21.77062

This refitted model often improves prediction accuracy relative to the original (biased) lasso estimator while retaining the sparsity pattern discovered by the \(\ell_1\) penalty.