3.4 The Student Performance Example

Up to now, we studied two complementary approaches to controlling model complexity in linear regression. Classical subset-selection methods (best-subset, forward, backward, and stepwise) explicitly search over candidate models and penalize complexity via information criteria such as AIC, BIC, adjusted \(R^2\), or Mallows’ \(C_p\).

When the number of predictors is large, continuous shrinkage methods become preferable. Ridge regression (\(\ell_2\) penalty) produces a closed-form estimator that shrinks all coefficients toward zero, with stronger shrinkage applied to low-variance principal-component directions; it is especially useful under multicollinearity but does not perform variable selection. Lasso regression (\(\ell_1\) penalty) simultaneously shrinks coefficients and sets some of them exactly to zero, thereby combining estimation and variable selection; it is particularly effective when the underlying signal is sparse or when \(p>n\). Both ridge and lasso require the predictors to be centered and scaled, and the tuning parameter \(\lambda\) is typically chosen by cross-validation.

The choice between these methods ultimately depends on the bias–variance trade-off and on whether sparsity is expected in the true model.

 

In this final section we illustrate the practical implementation of the variable-selection and shrinkage methods developed earlier, using a real data example.

The analysis is based on the student-mat.csv data set, which contains information on student achievement in secondary education at two Portuguese schools. The original study is described in:

“Using data mining to predict secondary school student performance” By P. Cortez, A. M. G. Silva. 2008. Published in Proceedings of 5th Annual Future Business Technology Conference.

The data are publicly available from the UCI Machine Learning Repository: here

The analysis of this data set is available in R and Python.