2.8 Categorical Variables in MLR

As we discussed, some of the predictors may be categorical. Consider a categorical predictor with \(k\) levels. When it is included in a linear regression model it is represented it is represented by binary indicator variables. Two common coding schemes are used:

One-hot encoding creates $k $ binary indicators, one for each level: \[D_i = \begin{cases} 1 & \text{if the observation belongs to level } i, \\ 0 & \text{otherwise}. \end{cases}\]

Dummy coding (the default in classical linear regression) creates only $ k-1 $ binary indicators and absorbs one level into the intercept. The indicators take the form \[D_i = \begin{cases} 0 & \text{if the observation is not at level } i, \\ 1 & \text{if the observation is at level } i, \end{cases}\] where the omitted level is called the reference level.

An Example with Categorical Variables

Consider a categorical predictor with three levels (\(a\), \(b\), \(c\)).

One-hot encoding produces \(k = 3\) columns: \[\begin{pmatrix} a \\ a \\ b \\ b \\ b \\ c \\ c \end{pmatrix} \quad\Longrightarrow\quad \begin{pmatrix} 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 1 \end{pmatrix}\]

Dummy coding (with level \(c\) as the reference) produces \(k-1 = 2\) columns: \[\begin{pmatrix} a \\ a \\ b \\ b \\ b \\ c \\ c \end{pmatrix} \quad\Longrightarrow\quad \begin{pmatrix} 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 1 \\ 0 & 1 \\ 0 & 0 \\ 0 & 0 \end{pmatrix}\]

In the dummy-coding matrix, Column 1 is the indicator for level \(a\) and Column 2 is the indicator for level \(b\). Level \(c\) is absorbed into the intercept.

 

In general any level may be chosen as the reference level; the choice is usually driven by subject-matter context rather than by statistical considerations.

Dummy coding vs. One-hot encoding

  • Dummy coding uses \(k-1\) indicators and keeps the design matrix full rank when an intercept is present. This is the standard approach in classical linear regression.

  • One-hot encoding uses \(k\) indicators. When an intercept is also included, the \(k\) columns sum to the constant vector \(\mathbf{1}\), creating perfect multicollinearity; \(\mathbf{X}^T\mathbf{X}\) becomes singular and the ordinary least-squares estimator is no longer unique.

  • Fitted and predicted values are identical under both codings. Only the interpretation of the individual coefficients changes. In practice, any level may be chosen as the reference; the choice is usually guided by subject-matter context rather than by statistical considerations.