Chapter 4 NonLinear Regression

As we discussed in the previous sections, multiple linear regression relies on a set of assumptions about the error terms (normality, homoscedasticity, and independence) together with the fundamental assumption that the mean response is a linear function of the predictors. When one or more of these assumptions are violated, the applicability of the linear model is limited. In this section we focus on relaxing the linearity assumption and introduce three of the most widely used approaches to nonlinear regression:

  1. Polynomial regression
  2. Regression splines
  3. Smoothing splines

   

A Note on Nonlinearity

Assume that the true underlying model takes the form

\[Y = f(\mathbf{X}) + \varepsilon\]

where \(\varepsilon\) satisfies the usual assumptions. So far we have approximated the unknown function \(f\) with a multiple linear regression model:

\[Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} +\ldots + \beta_p X_{ip} +\varepsilon_i \]

This model is linear both in the predictors \(X_{ij}\) and in the coefficients \(\beta_j\).

Now consider the following two models: \[Y_i = \beta_1 X_{i1} + \beta_2 X_{i1}^2 +\beta_3 X_{i2} + \beta_4 X_{i2}^2 + \beta_5 X_{i1} X_{i2} +\varepsilon_i \] or \[\log_{10}Y_i = \beta_1 X_{i1} + \beta_2 \sqrt{X_{i1}} +\beta_3 e^{X_{i3}} +\varepsilon_i\]

Both are nonlinear in the predictors but remain linear in the coefficients \(\beta_j\). In contrast, the model \[Y_i = \frac{\gamma_0}{1+ \gamma_1 e^{\gamma_2 X_i}} + \varepsilon_i \] is nonlinear in both the parameters and the predictors.

In this chapter we approximate the general nonlinear relationship \(f\) using models that are nonlinear in the predictors (so that we can capture nonlinear associations between the response and the features) yet linear in the coefficients. Equivalently, these models can be viewed as ordinary linear regressions in which the original predictors have been replaced by nonlinear transformations of those predictors.

The main advantage of this approach is that we can often describe nonlinear relationships effectively while still estimating the parameters easily. Because the models remain linear in the parameters, the coefficients continue to be obtained by ordinary least squares, which again reduces to solving a system of linear equations.