1.2 Supervised Learning Framework

In this class, our main focus is supervised learning. This means that we observe target (outcome, response) variable \(Y\) that we wish to predict using a set of features (predictors), \(X\). We assume that the relationship between the response \(Y\) and the features is given by a function \(f\) that is known up to a parameter vector \(w\): \[\mathbf{Y} = f(\mathbf{X}, w) + varepsilon \] When \(f\) is fully known except for the value of \(w\) (for example when \(f\) is linear), then learning problem reduces to estimating \(w\). To do so, we collect a training sample \(\bigl\{\mathbf{x}_i, \mathbf{y}_i \bigr\}_{i=1}^{n}\) and minimize an objective function_with respect to \(w\): \[F(w) = \sum_{i=1}^{n} loss \Bigl(y_i,\, f \bigl(\mathbf{x}_i, w \bigr) \Bigr)\] The loss function measures the discrepancy between the model prediction \(f(\mathbf{x}_i, w)\) and the observed outcome \(\mathbf{y}_i\). The choice of loss is made by the practitioner; the most common examples are:

  • Squared Error Loss \[\text{loss} = \Bigl[ y_i - f\bigl(x_i; w \bigr)\Bigr]^2\]

  • Absolute Error Loss \[\text{loss} = \Bigl| y_i - f\bigl(x_i; w \bigr)\Bigr|\]

  • Log Loss (binary classification) \[\text{loss} = -y_i \log f(x_i; w) - (1-y_i)\log\bigl(1 - f(x_i; w)\bigr) \]

  • 0-1 Loss \[\text{loss} =\mathbf{1}_{\{y_i \neq \hat{y}_i\}} \quad\text{where}\quad \hat{y}_i = f(x_i; w) \]

Depending on the chosen loss, the minimizer of \(F(w)\) may or may not admit a closed-form expression. When a closed form is unavailable we resort to numerical optimization algorithms. In many cases gradient-based methods (in particular gradient descent and its variants) can be used to obtain a good solution.