2.8 Categorical Variables in MLR
As we discussed, some of the predictors may be categorical. Consider a categorical predictor with \(k\) levels. When it is included in a linear regression model it is represented it is represented by binary indicator variables. Two common coding schemes are used:
One-hot encoding creates $k $ binary indicators, one for each level: \[D_i = \begin{cases} 1 & \text{if the observation belongs to level } i, \\ 0 & \text{otherwise}. \end{cases}\]
Dummy coding (the default in classical linear regression) creates only $ k-1 $ binary indicators and absorbs one level into the intercept. The indicators take the form \[D_i = \begin{cases} 0 & \text{if the observation is not at level } i, \\ 1 & \text{if the observation is at level } i, \end{cases}\] where the omitted level is called the reference level.
An Example with Categorical Variables
Consider a categorical predictor with three levels (\(a\), \(b\), \(c\)).
One-hot encoding produces \(k = 3\) columns: \[\begin{pmatrix} a \\ a \\ b \\ b \\ b \\ c \\ c \end{pmatrix} \quad\Longrightarrow\quad \begin{pmatrix} 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 1 \end{pmatrix}\]
Dummy coding (with level \(c\) as the reference) produces \(k-1 = 2\) columns: \[\begin{pmatrix} a \\ a \\ b \\ b \\ b \\ c \\ c \end{pmatrix} \quad\Longrightarrow\quad \begin{pmatrix} 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 1 \\ 0 & 1 \\ 0 & 0 \\ 0 & 0 \end{pmatrix}\]
In the dummy-coding matrix, Column 1 is the indicator for level \(a\) and Column 2 is the indicator for level \(b\). Level \(c\) is absorbed into the intercept.
In general any level may be chosen as the reference level; the choice is usually driven by subject-matter context rather than by statistical considerations.
Dummy coding vs. One-hot encoding
Dummy coding uses \(k-1\) indicators and keeps the design matrix full rank when an intercept is present. This is the standard approach in classical linear regression.
One-hot encoding uses \(k\) indicators. When an intercept is also included, the \(k\) columns sum to the constant vector \(\mathbf{1}\), creating perfect multicollinearity; \(\mathbf{X}^T\mathbf{X}\) becomes singular and the ordinary least-squares estimator is no longer unique.
Fitted and predicted values are identical under both codings. Only the interpretation of the individual coefficients changes. In practice, any level may be chosen as the reference; the choice is usually guided by subject-matter context rather than by statistical considerations.