2.10 Model Diagnostics
We made several assumptions when defining the linear regression model \[\mathbf{y} = \mathbf{X}\beta + \varepsilon,\,\, \text{ where } \,\, \varepsilon \sim IID(0, \sigma^2)\]
These assumptions are:
Linearity: the conditional mean of \(Y\) is a linear function of the predictors.
Constant Variance(homoscedasticity): all responses share a common, constant variance \(\sigma^2\).
Uncorrelated errors: the errors \(\varepsilon_i\) and \(\varepsilon_j\) (and therefore \(Y_i\) and \(Y_j\)) are uncorrelated for \(i\neq j\).
(Normality): this is not required for the derivation of the least-squares estimators themselves. It becomes necessary, however, when we want exact finite-sample distributional results for the estimators, fitted values, and residuals, and it is especially important for hypothesis testing.
If the goal is to interpret the effects of the predictors on the response (rather than pure prediction), all of the above assumptions should be checked with diagnostic plots and tests, and appropriate remedial measures should be taken when they are violated.
Because the primary focus of this course is prediction, we refer students to a dedicated regression course or textbook for a thorough treatment of diagnostics and remedial measures in multiple linear regression.
Outliers
Outliers can be a concern in both large and small data sets. When the sample size is modest, it is advisable to test formally for outliers—for example, with tests based on the leave-one-out prediction error. In very large data sets such tests lose power and are less useful; one should instead adjust for multiple comparisons or adopt alternative strategies.
If outliers appear to be problematic, useful practical steps include:
examining the range of each variable,
applying a log, square-root, or other suitable transformation to right-skewed predictors or to the response \(Y\), and
using winsorization to limit the influence of extreme values.