Student Performance Example
In this final section we illustrate the practical implementation of the variable-selection and shrinkage methods developed earlier, using a real data example.
The analysis is based on the student-mat.csv data set,
which contains information on student achievement in secondary education
at two Portuguese schools. The original study is described in:
Cortez, P., & Silva, A. M. G. (2008). Using data mining to predict secondary school student performance. Proceedings of the 5th Annual Future Business Technology Conference.
The data are publicly available from the UCI Machine Learning
Repository:
Student
Performance data set.
The variables in the data set are the following:
school – student’s school (binary:
GP – Gabriel Pereira or MS – Mousinho da
Silveira)sex – student’s sex (binary: F –
female or M – male)age – student’s age (numeric: from 15 to
22)address – home address type (binary:
U – urban or R – rural)famsize – family size (binary:
LE3 – ≤ 3 or GT3 – > 3)Pstatus – parent’s cohabitation status
(binary: T – living together or A –
apart)Medu – mother’s education (numeric: 0 – none,
1 – primary (4th grade), 2 – 5th–9th grade, 3 – secondary, 4 – higher
education)Fedu – father’s education (numeric: same
coding as Medu)Mjob – mother’s occupation (nominal:
teacher, health, services,
at_home, other)Fjob – father’s occupation (nominal: same
categories as Mjob)reason – reason for choosing the school
(nominal: home, reputation,
course, other)guardian – student’s guardian (nominal:
mother, father, other)traveltime – home-to-school travel time
(numeric: 1 – <15 min, 2 – 15–30 min, 3 – 30–60 min, 4 –
>1 hour)studytime – weekly study time (numeric: 1 –
<2 h, 2 – 2–5 h, 3 – 5–10 h, 4 – >10 h)failures – number of past class failures
(numeric: n if 1 ≤ n < 3, else 4)schoolsup – extra educational support (binary:
yes/no)famsup – family educational support (binary:
yes/no)paid – extra paid classes in the course subject
(binary: yes/no)activities – extra-curricular activities
(binary: yes/no)nursery – attended nursery school (binary:
yes/no)higher – wants to take higher education
(binary: yes/no)internet – Internet access at home (binary:
yes/no)romantic – in a romantic relationship (binary:
yes/no)famrel – quality of family relationships
(numeric: 1 – very bad to 5 – excellent)freetime – free time after school (numeric: 1
– very low to 5 – very high)goout – going out with friends (numeric: 1 –
very low to 5 – very high)Dalc – workday alcohol consumption (numeric: 1
– very low to 5 – very high)Walc – weekend alcohol consumption (numeric: 1
– very low to 5 – very high)health – current health status (numeric: 1 –
very bad to 5 – very good)absences – number of school absences (numeric:
0 to 93)G1 – first-period grade (numeric: 0 to
20)G2 – second-period grade (numeric: 0 to
20)G3 – final grade (numeric: 0 to 20; the
response variable)stu_performance0 <- read.csv2("student-mat.csv", header = TRUE)
Our goal is to predict the final grade (G3) using the
penalized regression methods studied in this chapter. Because standard
ridge and lasso do not handle categorical predictors directly (a
group-lasso formulation would be required), we restrict attention to the
numeric predictors only.
## Keep only numeric predictors
stu_performance <- stu_performance0[, -c(1, 2, 4, 5, 6, 9, 10, 11, 12, 16, 17, 18, 19, 20, 21, 22, 23, 31, 32)]
## Rename the response
names(stu_performance)[14] <- "Grades"
head(stu_performance)
## age Medu Fedu traveltime studytime failures famrel freetime goout Dalc Walc
## 1 18 4 4 2 2 0 4 3 4 1 1
## 2 17 1 1 1 2 0 5 3 3 1 1
## 3 15 1 1 1 2 3 4 3 2 2 3
## 4 15 4 2 1 3 0 3 2 2 1 1
## 5 16 3 3 1 2 0 4 3 2 1 2
## 6 16 4 3 1 2 0 5 4 2 1 2
## health absences Grades
## 1 3 6 6
## 2 3 4 6
## 3 3 10 10
## 4 5 2 15
## 5 5 4 10
## 6 5 10 15
n <- nrow(stu_performance) # sample size
p <- ncol(stu_performance) - 1 # number of predictors (excluding intercept)
We begin by fitting the full linear model that includes every numeric predictor:
performance.full = lm(Grades ~ ., data=stu_performance)
summary(performance.full)
##
## Call:
## lm(formula = Grades ~ ., data = stu_performance)
##
## Residuals:
## Min 1Q Median 3Q Max
## -12.2769 -2.1753 0.4053 2.7927 8.7122
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 13.53702 3.33724 4.056 6.05e-05 ***
## age -0.25623 0.17880 -1.433 0.15268
## Medu 0.58494 0.25650 2.280 0.02313 *
## Fedu -0.09135 0.25465 -0.359 0.72000
## traveltime -0.40184 0.31439 -1.278 0.20197
## studytime 0.26583 0.26903 0.988 0.32373
## failures -1.83146 0.31201 -5.870 9.49e-09 ***
## famrel 0.28018 0.24537 1.142 0.25422
## freetime 0.34139 0.23096 1.478 0.14021
## goout -0.62638 0.22179 -2.824 0.00499 **
## Dalc -0.13723 0.32034 -0.428 0.66862
## Walc 0.35312 0.23834 1.482 0.13928
## health -0.17429 0.15620 -1.116 0.26521
## absences 0.03197 0.02760 1.158 0.24745
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.211 on 381 degrees of freedom
## Multiple R-squared: 0.183, Adjusted R-squared: 0.1551
## F-statistic: 6.564 on 13 and 381 DF, p-value: 2.525e-11
A simple (but inefficient) approach to model selection is to compare the full model with a reduced model that retains only the apparently significant variables, using a partial F-test:
performance.red = lm(Grades ~ failures + goout + Medu, data=stu_performance)
## Model comparison based on a partial F test
anova(performance.red, performance.full)
## Analysis of Variance Table
##
## Model 1: Grades ~ failures + goout + Medu
## Model 2: Grades ~ age + Medu + Fedu + traveltime + studytime + failures +
## famrel + freetime + goout + Dalc + Walc + health + absences
## Res.Df RSS Df Sum of Sq F Pr(>F)
## 1 391 6956.9
## 2 381 6756.7 10 200.22 1.129 0.339
In this particular case we fail to reject the null hypothesis, so the reduced model is preferred by the F-test. However, this ad-hoc strategy is neither systematic nor optimal. We therefore turn to formal search algorithms that explore the model space while applying an explicit complexity penalty.
leaps package
library(leaps)
regsubsets_model <- regsubsets(Grades ~ ., data = stu_performance, nvmax = p)
rs <- summary(regsubsets_model)
The object rs stores the information criteria for every
model examined. By default regsubsets considers models of
size up to 8; we have overridden this limit with nvmax = p
so that all model sizes from 1 to 13 are evaluated.
Below we extract the \(R^2\), Adjusted \(R^2\), \(AIC\), \(BIC\) and \(C_p\)-Mallows criteria for the 13 models that have been evaluated:
## Extract the models' R^2
rs$rsq
## [1] 0.1298989 0.1483134 0.1587670 0.1633698 0.1668323 0.1696634 0.1724079
## [8] 0.1746913 0.1773575 0.1800384 0.1823149 0.1827012 0.1829771
The best model is the 13th model (as expected).
## Extract the models' Adjusted R^2
rs$adjr2
## [1] 0.1276849 0.1439681 0.1523125 0.1547890 0.1561232 0.1568231 0.1574385
## [8] 0.1575865 0.1581269 0.1586853 0.1588305 0.1570269 0.1550997
The best model is the 11th model.
## Extract the models' Cp Mallows
rs$cp
## [1] 14.751803 8.164618 5.289832 5.143431 5.528772 6.208548 6.928720
## [8] 7.863899 8.620578 9.370388 10.308796 12.128682 14.000000
The best model is the 4th model.
## Extract the models' BIC
rs$bic
## [1] -43.004859 -45.475340 -44.374670 -40.562949 -36.222213 -31.587827
## [7] -26.916684 -22.029154 -17.328400 -12.638892 -7.758187 -1.965927
## [13] 3.879571
The best model is the 2nd model.
The matrix rs$which indicates which variables appear in
each model (TRUE = included):
rs$which
## (Intercept) age Medu Fedu traveltime studytime failures famrel freetime
## 1 TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE
## 2 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE
## 3 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE
## 4 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE
## 5 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE
## 6 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE
## 7 TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE
## 8 TRUE TRUE TRUE FALSE TRUE FALSE TRUE FALSE TRUE
## 9 TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE
## 10 TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE
## 11 TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE
## 12 TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE TRUE
## 13 TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE
## goout Dalc Walc health absences
## 1 FALSE FALSE FALSE FALSE FALSE
## 2 FALSE FALSE FALSE FALSE FALSE
## 3 TRUE FALSE FALSE FALSE FALSE
## 4 TRUE FALSE FALSE FALSE FALSE
## 5 TRUE FALSE FALSE FALSE FALSE
## 6 TRUE FALSE TRUE FALSE FALSE
## 7 TRUE FALSE FALSE FALSE TRUE
## 8 TRUE FALSE TRUE FALSE TRUE
## 9 TRUE FALSE TRUE TRUE FALSE
## 10 TRUE FALSE TRUE TRUE TRUE
## 11 TRUE FALSE TRUE TRUE TRUE
## 12 TRUE TRUE TRUE TRUE TRUE
## 13 TRUE TRUE TRUE TRUE TRUE
For example, Model 1 only contains failures, and the
intercept.
We can also compute AIC and BIC “by hand” for the sequence of models:
msize <- 1:p
par(mfrow = c(1, 2))
Aic <- n * log(rs$rss / n) + 2 * msize
Bic <- n * log(rs$rss / n) + msize * log(n)
Aic
## [1] 1148.427 1141.978 1139.100 1138.932 1139.294 1139.950 1140.642 1141.551
## [9] 1142.272 1142.983 1143.885 1145.698 1147.565
AIC chooses the 4th model.
Bic
## [1] 1152.406 1149.936 1151.036 1154.848 1159.189 1163.823 1168.494 1173.382
## [9] 1178.082 1182.772 1187.653 1193.445 1199.290
As we saw before, BIC selects the 2nd model. We can also plot the results for the various models. So, we have
par(mfrow = c(2, 2))
plot(msize, rs$adjr2, xlab = "Number of parameters", ylab = "Adjusted R²")
plot(msize, rs$cp, xlab = "Number of parameters", ylab = "Mallows’ Cp")
plot(msize, Aic, xlab = "Number of parameters", ylab = "AIC")
plot(msize, Bic, xlab = "Number of parameters", ylab = "BIC")
As is typical, AIC selects a larger model than BIC. Although the BIC-optimal model does not attain the absolute minimum AIC, its AIC value is very close to the optimum.
cbind(rs$which[which.min(Aic),], rs$which[which.min(Bic), ])
## [,1] [,2]
## (Intercept) TRUE TRUE
## age FALSE FALSE
## Medu TRUE TRUE
## Fedu FALSE FALSE
## traveltime FALSE FALSE
## studytime FALSE FALSE
## failures TRUE TRUE
## famrel FALSE FALSE
## freetime TRUE FALSE
## goout TRUE FALSE
## Dalc FALSE FALSE
## Walc FALSE FALSE
## health FALSE FALSE
## absences FALSE FALSE
(Note that the BIC reported by leaps differs from our hand calculation by a constant; the ranking of models is therefore identical.)
cbind(rs$bic, Bic, rs$bic - Bic)
## Bic
## [1,] -43.004859 1152.406 -1195.411
## [2,] -45.475340 1149.936 -1195.411
## [3,] -44.374670 1151.036 -1195.411
## [4,] -40.562949 1154.848 -1195.411
## [5,] -36.222213 1159.189 -1195.411
## [6,] -31.587827 1163.823 -1195.411
## [7,] -26.916684 1168.494 -1195.411
## [8,] -22.029154 1173.382 -1195.411
## [9,] -17.328400 1178.082 -1195.411
## [10,] -12.638892 1182.772 -1195.411
## [11,] -7.758187 1187.653 -1195.411
## [12,] -1.965927 1193.445 -1195.411
## [13,] 3.879571 1199.290 -1195.411
To examine the second- and third-best models we request the three best models of each size:
regsubsets_model.1 = regsubsets(Grades ~ ., data=stu_performance, nbest = 3, nvmax = p)
rs.1 = summary(regsubsets_model.1)
#rs.1$which
msize.1 = apply(rs.1$which, 1, sum) - 1
par(mfrow=c(1,2))
Aic.1 = n*log(rs.1$rss/n) + 2*msize.1
Bic.1 = n*log(rs.1$rss/n) + msize.1*log(n)
plot(msize.1, Aic.1, xlab="No. of Parameters", ylab = "AIC")
plot(msize.1, Bic.1, xlab="No. of Parameters", ylab = "BIC")
par(mfrow=c(1,2))
plot(msize.1, Aic.1, ylim = c(1135, 1152), xlab="No. of Parameters", ylab = "AIC")
plot(msize.1, Bic.1, ylim = c(1140, 1230), xlab="No. of Parameters", ylab = "BIC")
## Top three models by AIC
rs.1$which[order(Aic.1)[1:3], ]
## (Intercept) age Medu Fedu traveltime studytime failures famrel freetime
## 4 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE TRUE
## 3 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE
## 5 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE TRUE
## goout Dalc Walc health absences
## 4 TRUE FALSE FALSE FALSE FALSE
## 3 TRUE FALSE FALSE FALSE FALSE
## 5 TRUE FALSE FALSE FALSE FALSE
## Top three models by BIC
rs.1$which[order(Bic.1)[1:3], ]
## (Intercept) age Medu Fedu traveltime studytime failures famrel freetime
## 2 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE
## 3 TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE
## 1 TRUE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE
## goout Dalc Walc health absences
## 2 FALSE FALSE FALSE FALSE FALSE
## 3 TRUE FALSE FALSE FALSE FALSE
## 1 FALSE FALSE FALSE FALSE FALSE
step Fucntion
The step function implements forward, backward, and
stepwise search. By default it uses AIC; setting k = log(n)
yields BIC.
## Stepwise Selection with AIC
stepAIC = step(performance.full, trace=0, direction="both")
#stepAIC = step(performance.full, direction="both")
## Stepwise Selection with BIC
n = dim(stu_performance)[1]
stepBIC = step(performance.full, trace=0, direction="both", k=log(n))
The selected variables can be extracted as follows:
sel.var.AIC = attr(stepAIC$terms, "term.labels")
sel.var.BIC = attr(stepBIC$terms, "term.labels")
sel.var.AIC
## [1] "Medu" "failures" "freetime" "goout"
length(sel.var.AIC)
## [1] 4
length(sel.var.BIC)
## [1] 2
c("Medu", "failures","freetime", "goout") %in% sel.var.AIC
## [1] TRUE TRUE TRUE TRUE
sel.var.BIC %in% sel.var.AIC
## [1] TRUE TRUE
The package glmnet implements both ridge (\(\alpha = 0\)) and lasso (\(\alpha = 1\)). Additional examples can be
found at https://web.stanford.edu/~hastie/glmnet/glmnet_beta.html
library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 5.0
## The input in the glmnet function should be in matrix format:
X = as.matrix(stu_performance[, names(stu_performance) != "Grades"])
Y = stu_performance$Grades
We randomly split the data into an 80 % training set and a 20 % test set. Model selection and coefficient estimation are performed on the training data; predictive performance is evaluated on the held-out test set.
## Training/ Testing Data Sets
set.seed(598) # for reproducibility
ntest <- round(n * 0.2)
ntrain <- n - ntest
test.id <- sample(1:n, ntest)
Ytest <- Y[test.id]
For reference we also fit the ordinary least-squares model on the training data and compute its test-set MSE:
full.model <- lm(Y[-test.id] ~ ., data = stu_performance[-test.id, names(stu_performance) != "Grades"])
Ytest.pred <- predict(full.model, newdata = stu_performance[test.id, names(stu_performance) != "Grades"])
mean((Ytest - Ytest.pred)^2)
## [1] 22.31678
Fit a set of ridge regression models using glmnet with
the default \(\lambda\) sequence.
Selecting \(\alpha=0\) to run a
ridge
We plot the fitted output of the ridge regression. We can use the
xvar argument to decide what is plotted on the x-axis. We
can also label the curves with the variable index numbers simply by
setting label = TRUE.
ridge.lm = glmnet(X[-test.id, ], Y[-test.id], alpha = 0)
plot(ridge.lm , label = TRUE, xvar = "lambda")
We can obtain the \(\lambda\) values
length(ridge.lm$lambda)
## [1] 100
## remove the length function to see the lambda values
and the coefficients :
dim(ridge.lm $beta) ## coefficients for non-intercept predictors
## [1] 13 100
length(ridge.lm $a0) ## intercept
## [1] 100
## You can also retrieve coefficients (including intercept) using coef(myridge)
dim(coef(ridge.lm ))
## [1] 14 100
## The two coefficient matrices should be the same
sum((coef(ridge.lm ) - rbind(ridge.lm $a0, ridge.lm $beta))^2)
## [1] 0
How are the intercepts computed?
k = 2;
## Compute the "intercept" by hand:
mean.ridge = apply(X[-test.id, ], 2, mean) ## mean vector for training X
mean(Y[-test.id]) - sum(mean.ridge * ridge.lm $beta[, k])
## [1] 10.60974
## Extract intercept from the estimated coefficients
ridge.lm$a0[k] ## intercept for lambda = myridge$lambda[k]
## s1
## 10.60974
## Both approaches give the same value
sum((mean(Y[-test.id]) - mean.ridge %*% ridge.lm $beta - ridge.lm $a0)^2)
## [1] 1.072851e-27
We can also choose xvar="dev" to plot the fitted
coefficients against the percent deviance. This is the percent of
deviance explained on the training data, and is a measure of complexity
of the model. We can see that the coefficients do not change much after
a point the but the coefficients start blowing up a bit.
plot(ridge.lm, xvar = "dev", label = TRUE)
If we want to use cross-validation to choose \(\lambda\), then we can use the function
cv.glmnet. This function is using a \(K\)-fold cross validation method to obtain
the “optimal lambda”. The cv.glmnet has the attribute
nfolds to select the number of folds we prefer, the option
foldid to specify our own folds, and
type.measure to choose the loss function
(deviance or mse for squared loss, and
mae for mean absolute error).
ridge.cv = cv.glmnet(X[-test.id, ], Y[-test.id], alpha = 0)
## the default in nfolds is 10
Some basic output information can be obtained
print(ridge.cv)
##
## Call: cv.glmnet(x = X[-test.id, ], y = Y[-test.id], alpha = 0)
##
## Measure: Mean-Squared Error
##
## Lambda Index Measure SE Nonzero
## min 1.50 75 17.37 1.639 13
## 1se 46.96 38 19.01 2.014 13
As a side note, the cv.glmnet function allows parallel
computing for large scale problems.
The results are stored in
ridge.cv$cvm mean CV errors ridge.cv$cvsd:
standard errors of CVm
We can also plot the result of the cv.glmnet:
plot(ridge.cv)
The CV performance continues to drop when \(\lambda\) gets smaller. So, we provide
R with a \(\lambda\)
sequence dense in \([-2, 2]\):
lambda.seq = exp(seq(-2, 4, length=100))
ridge.cv.seq = cv.glmnet(X[-test.id,], Y[-test.id], alpha=0, lambda=lambda.seq)
plot(ridge.cv.seq)
We can still use the coef and predict
methods for cv.glmnet objects in the same way as for the
glmnet. In this case, we also have two special strings
supported by \(s\) (the values of \(\lambda\) requested):
lambda.min: the value of lambda that achieves the
minimum MSE (i.e. cvm – the left dashed vertical
line)
lambda.1se: the largest value of lambda (i.e., the
largest regularization, the smallest df) whose MSE
(i.e. cvm) is within one-standard-error of the
cvm of lambda.min – i.e. the smallest MSE (–
the left dashed vertical line).
Note: by definition lambda.1se (right
dashed vertical line) is always bigger than or equal to
lambda.min (left dashed vertical line).
Check how lambda.min and lambda.1se are
computed.
names(ridge.cv.seq)
## [1] "lambda" "cvm" "cvsd" "cvup" "cvlo"
## [6] "nzero" "call" "name" "glmnet.fit" "lambda.min"
## [11] "lambda.1se" "index"
## Find lambda.min
ridge.cv.seq$lambda.min
## [1] 1.833195
ridge.cv.seq$lambda[which.min(ridge.cv.seq$cvm)]
## [1] 1.833195
## Find lambda.1se
ridge.cv.seq$lambda.1se
## [1] 54.59815
tmp.id = which.min(ridge.cv.seq$cvm)
max(ridge.cv.seq$lambda[ridge.cv.seq$cvm < ridge.cv.seq$cvm[tmp.id] + ridge.cv.seq$cvsd[tmp.id]])
## [1] 54.59815
Next we apply the two ridge regression models—one with
lambda.min and one with lambda.1se on a new
test data set and report the corresponding mean squared
prediction errors:
ridge.lambda.pred = glmnet(X[-test.id,], Y[-test.id], alpha=0, lambda = lambda.seq)
## Using the lambda.min
Ytest.ridge.min.pred = predict(ridge.lambda.pred, s = ridge.cv.seq$lambda.min, newx=X[test.id,])
mean((Ytest.ridge.min.pred - Y[test.id])^2)
## [1] 22.53208
## Using lambda.1se
Ytest.ridge.1se.pred = predict(ridge.lambda.pred, s = ridge.cv.seq$lambda.1se, newx=X[test.id,])
mean((Ytest.ridge.1se.pred - Y[test.id])^2)
## [1] 25.47849
If we want to use LASSO instead of Ridge, we need to set \(\alpha=1\) in the glmnet
function. In this case, we have
lasso.lm = glmnet(X[-test.id,], Y[-test.id], alpha = 1)
Using cross validation, we can see the CV path plot:
par(mfrow = c(1, 2))
plot(lasso.lm, label=TRUE, xvar = "norm") ## norm is the default value for xvar
plot(lasso.lm, label=TRUE, xvar = "lambda")
par(mfrow=c(1,1))
We can extract the number of non-zero coefficients for each value of \(\lambda\).
lasso.lm$df
## [1] 0 1 1 1 1 2 2 2 2 2 3 3 4 4 4 4 4 4 5 6 6 6 6 7 7
## [26] 7 7 7 7 7 7 7 7 7 9 9 9 9 9 9 9 9 9 10 12 12 12 12 12 12
## [51] 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
As with the ridge regression example, we can still use
cross-validation by calling the cv.glmnet:
lasso.cv = cv.glmnet(X[-test.id, ], Y[-test.id], alpha = 1)
plot(lasso.cv)
From the cv.glmnet object we can still compute
lambda.min and lambda.1se in the same way as
before:
## Computing lambda.min
lasso.cv$lambda.min
## [1] 0.1190689
## Computing lambda.1se
lasso.cv$lambda.1se
## [1] 0.9219072
We can also extract the LASSO coefficients:
lasso.min.pred.coef = predict(lasso.lm, s=lasso.cv$lambda.min, type="coefficients");
lasso.1se.pred.coef = predict(lasso.lm, s=lasso.cv$lambda.1se, type="coefficients");
cbind(lasso.min.pred.coef, lasso.1se.pred.coef);
## 14 x 2 sparse Matrix of class "dgCMatrix"
## s=0.1190689 s=0.9219072
## (Intercept) 14.920661406 10.75737536
## age -0.281420530 .
## Medu 0.600006298 0.02508545
## Fedu . .
## traveltime . .
## studytime . .
## failures -1.540566778 -0.76873824
## famrel . .
## freetime 0.254866691 .
## goout -0.422725540 .
## Dalc . .
## Walc . .
## health -0.103790831 .
## absences 0.008985155 .
## number of variables selected (including the intercept)
sum(lasso.1se.pred.coef != 0)
## [1] 3
sum(lasso.min.pred.coef != 0)
## [1] 8
## names of selected non-intercept variables
row.names(lasso.1se.pred.coef)[which(lasso.1se.pred.coef != 0)[-1]]
## [1] "Medu" "failures"
row.names(lasso.min.pred.coef)[which(lasso.min.pred.coef != 0)[-1]]
## [1] "age" "Medu" "failures" "freetime" "goout" "health" "absences"
Apply the two fitted models for prediction on test data:
lasso.lambda.pred = glmnet(X[-test.id, ], Y[-test.id], alpha = 1)
## Using the lambda.min
Ytest.lasso.min.pred = predict(lasso.lambda.pred, s = lasso.cv$lambda.min, newx = X[test.id,])
mean((Ytest.lasso.min.pred - Y[test.id])^2)
## [1] 22.301
## Using lambda.1se
Ytest.lasso.1se.pred = predict(lasso.lambda.pred, s = lasso.cv$lambda.1se, newx=X[test.id,])
mean((Ytest.lasso.1se.pred - Y[test.id])^2)
## [1] 23.71601
A common post-processing step is to refit an ordinary least-squares
model on the variables selected by the lasso (here using
lambda.1se) and then use that unbiased estimator for
prediction:
lasso.coef.1se = predict(lasso.lambda.pred, s = lasso.cv$lambda.1se, type="coefficients")
variables.lasso = row.names(lasso.coef.1se)[which(lasso.coef.1se != 0)[-1]]
variables.lasso
## [1] "Medu" "failures"
tmp.X <- X[, colnames(X) %in% variables.lasso, drop = FALSE]
train.df <- data.frame(Y = Y[-test.id], tmp.X[-test.id, , drop = FALSE])
lasso.refit <- coef(lm(Y ~ ., data = train.df))
# prediction on the test set
Ytest.pred <- lasso.refit[1] + as.matrix(tmp.X[test.id, , drop = FALSE]) %*% lasso.refit[-1]
mean((Ytest.pred - Y[test.id])^2)
## [1] 21.77062
This refitted model often improves prediction accuracy relative to the original (biased) lasso estimator while retaining the sparsity pattern discovered by the \(\ell_1\) penalty.