Chapter 5 Introduction to Neural Networks for Regression
In the previous chapter we discussed nonlinear regression in the sense that it is a model that is linear in the parameters. We used bases, polynomials, splines, and kernels for example to approximate a nonlinear relationship between the response and the predictors. We called this function \(f\) before, let’s further describe it as \[ f_\beta(x)=\beta^\top\phi(x) \]
for a feature map \(\phi\) chosen by us. Obviously, if the feature map \(\phi\) is wrong, then ridge, OLS or any other function will not be able to cover for the missing transformation.
In this chapter, we focus on \(\phi\). So, we keep the loss, the linear last layer, and learns \(\phi\). The resulting predictor is nonlinear in both \(\beta\) and \(x\). The algorithm that makes the gradient of \(J\) with respect to \(\beta\) cheap backpropagation.
In machine learning courses, it is common to introduce neural networks when one talks about classification, e.g., image or text classification. We are going to view it in the regression setting.
In our case, the response \(y\in\mathbb{R}\) is continuous, and it is scored by the MSE.