4.3 Smoothing Splines

In regression splines we must choose both the number and the locations of the knots (or, equivalently, the degrees of freedom). As we have seen, B-splines and natural cubic splines both construct an \(n\times p\) basis matrix \(\mathbf{F}\) and then fit a linear model on that basis. Inevitably this requires decisions about the order of the spline, the number of knots (often guided by AIC, BIC or cross-validation), and even the placement of those knots—tasks that can be quite challenging. It is therefore natural to ask whether there exists an alternative approach that selects the effective complexity of the fit automatically.

A smoothing spline is designed precisely to balance fidelity to the data against smoothness of the fitted curve. The construction begins with an extreme idea: place a knot at every distinct observed value \(x_1,\dots,x_n\). This produces a natural-cubic-spline basis of dimension \(n\), \[\mathbf{y}_{n\times 1} = \mathbf{F}_{n\times p} \beta_{p\times 1}\]

Such a basis can interpolate the data perfectly and therefore overfits. To control the overfitting we borrow the idea of penalization introduced earlier and minimize a ridge-type objective \[\min_{\beta} \Bigl\{ ||\mathbf{y}-\mathbf{F} \beta ||^2 + \lambda \beta^T \Omega \beta \Bigr\}\] where the tuning parameter \(\lambda\ge 0\) is typically chosen by cross-validation and the penalty matrix \(\Omega\) will be defined shortly. We now examine whether the solution of this penalized problem possesses attractive statistical properties.

 

4.3.1 The Roughness Penalty Approach

Let \(S[a,b]\) be the space of all ``smooth’’ functions defined on \([a,b]\). This is a second order Sobolev space where global polynomial functions and cubic splines functions live (\(S[a,b]\) is an infinite-dimensional function space. Our goal is to find the best function in \(S[a,b]\) to approximate \(f\).

Let us consider solving the following Penalized Residual Sum of Squares problem: \[ RSS_\lambda(g) = \sum_{i=1}^n [y_i - g(x_i)]^2 + \lambda \int_a^b [g''(x)]^2 dx. \] where \(\lambda\) is a smoothing parameter. The first term measures the closeness of the model to the data, while the second term penalizes the roughness/curvature of the function. We solve this problem on \([a,b]=[\min x_i, \max _i]\) for functions with finite roughness penalty \(\int_a^b [g''(x)]^2 dx < \infty\). This is known as a second order Sobolev space.

  Note that \(\int_a^b [g''(x)]^2 dx\) is called the roughness penalty.  

From the expression above, we can see that \(\lambda\) is the smoothing parameter that controls the bias-variance trade-off. Therefore, when \(\lambda=0\), we interpolate the data and that leads us to overfitting. When \(\lambda=\infty\), then we return to linear least-squares regression. It turns out that the solution to the penalized residual sum of squares has to be a NCS. Indeed,

Theorem \[\min_g RSS_{\lambda}(g) = \min_{\tilde{g}} RSS_{\lambda}(\tilde{g}) \]} where \(\tilde{g}\) is a NCS with knots at the unique \(n\) data points.

Let \(g(x)\) be the optimal solution. Since the loss part in \(RSS_\lambda(g)\) only involves \(n\) data points, we can find define a natural cubic spline (NCS) fit that we can call \(\tilde{g}(x)\) such that it matches \(g(x)\) at the observations \(x_i\), \(i=1, \ldots, n\), i.e. \[g(x_i) = \tilde{g}(x_i), i=1, \ldots, n\] We can always find such \(\tilde{g}\) since our space consists of \(n\) basis. Then, we can show that \[\int g^{''2} dx \geq \int \tilde{g}^{''2} dx\] meaning that we will always prefer the \(\tilde{g}\), the NCS ``representation’’ of \(g\), since the penalty is smaller, and the loss doesn’t change.

4.3.2 Proof

To establish the Theorem, we essentially need to show that \[\int g''^{2} dx \geq \int \tilde{g}''^{2} dx\]

Thus, we define \(h(x) = g(x) - \tilde{g}(x)\) for which we know that \(h(x_i)=0\) for \(i=1, \ldots, n\). Then \[ \int g''^2 \, dx = \int \tilde{g}''^2 \, dx + \int h''^2 \, dx + 2 \int \tilde{g}'' h'' \, dx \] and without loss of generality assuming that the \(x_i\)s are ordered, we obtain: \[\begin{align*} \int \tilde{g}'' h'' \, dx &= \tilde{g}'' h' \Big|_a^b - \int_a^b h' \tilde{g}^{(3)} \, dx\\ &= - \sum_{i=1}^{n-1} \tilde{g}^{(3)}\!\left(x_j^+\right) \int_{x_j}^{x_{j+1}} h' \, dx \quad (\tilde{g}^{(3)} \text{ constant piecewise})\\ &= - \sum_{i=1}^{n-1} \tilde{g}^{(3)}\!\left(x_j^+\right) \left(h(x_{j+1}) - h(x_j)\right) \end{align*}\] The second equation is because \(\tilde{g}\) is a NCS and therefore we know that it has zero second derivative on the two boundaries \(a\) and \(b\). The third equation is true, because \(\tilde{g}\) is at most a 3rd order polynomial on any region and as a consequence has constant third derivatives, which we can pull out of the integration. The last equation holds because we said that \(h(x)=0\) on all the observation points \(x_i\). Therefore, this shows that the roughness penalty of our NCS solution is no larger than the best solution \(g\). If we also take into account that \(\tilde{g}\) is also in the space \(S[a,b]\), then \(g\) must be our NCS solution.