4.4 Fitting Smoothing Splines
From the preceding discussion we conclude that the minimizer admits a finite-dimensional representation \[\hat{g}(x) = \sum_i \beta_i N_i(x)\] where the \(N_i\) are a natural-cubic-spline basis with knots placed at the unique observed values of $ x $. Substituting this expansion into the roughness penalty yields \[\begin{align*} \int_{a}^{b} g^{''2} dx &= \int \left( \sum_i \beta_i N_i^{''}(x)\right)^2 dx\\ &= \sum_{i,j} \beta_i \beta_j \int N_i^{''}(x) N_j^{''}(x) dx\\ &=\beta^T \Omega \beta \end{align*}\] where the \(n\times n\) penalty matrix has entries \[\Omega_{ij} = \int N^{''}_{i}(x) N^{''}_{j}(x) dx\]
Consequently the penalized residual sum of squares becomes a function of the coefficient vector alone:
\[RSS_{\lambda}(\beta) = (\mathbf{y} - \mathbf{F}\beta)^T(\mathbf{y} - \mathbf{F}\beta) + \lambda \beta^T \Omega \beta \]
This is a ridge-type criterion whose minimizer is given in closed form by
\[\hat{\beta} = \arg \min_{\beta} RSS_{\lambda}(\beta)\] \[=(\mathbf{F}^T \mathbf{F} +\lambda \Omega)^{-1} \mathbf{F}^T \mathbf{y}\]
The smoothing spline version of the ``hat’’ matrix is called the smoother matrix \[\mathbf{\hat{y}} = \mathbf{F} (\mathbf{F}^T \mathbf{F} +\lambda \Omega)^{-1} \mathbf{F}^T \mathbf{y} = S_{\lambda} \mathbf{y}\]
Remarks
We have done the analysis of degrees of freedom for ridge type regression. The degrees of freedom of a smoothing spline are \[df = tr(S_{\lambda})\] which ranges between \(0\) and \(n\). Under a special choice of basis (Demmler and Reinsch, 1975), a basis with double orthogonality property, can lead to \[\mathbf{F}^T \mathbf{F} = \mathbf{I}, \Omega = diag(d_i)\] where \(d_i\)s are arranges in an increasing order and in addition \(d_2=d_1=0\). Using this basis we have \[\hat{\beta} = (\mathbf{F}^T \mathbf{F} +\lambda \Omega)^{-1} \mathbf{F}^T \mathbf{y} = \Bigl(\mathbf{I} + \lambda diag(d_i) \Bigr)^{-1} \mathbf{F}^T \mathbf{y}\] i.e. \[\hat{\beta}_i = \frac{1}{1+\lambda d_i} \hat{\beta}_i^{LS}\]
Choosing \(\lambda\)
The smoothing parameter is selected by the same criteria used for ridge regression.
Leave-one-out CV: \[ \text{CV}(\lambda) = \frac{1}{n} \sum_{i=1}^n \left( \frac{y_i - \hat g(x_i)}{1 - S_\lambda(i,i)} \right)^2. \]
Generalized CV: \[ \text{GCV}(\lambda) = \frac{1}{n} \sum_{i=1}^n \left( \frac{y_i - \hat g(x_i)}{1 - \tfrac{1}{n}\text{tr}(S_\lambda)} \right)^2. \]