Chapter 9 Support Vector Machines

Support Vector Machines (SVMs) are a powerful approach to supervised learning, especially for classification. Rather than searching for any boundary that separates the classes, SVMs seek the boundary that maximizes the margin, that is the distance between the separating hyperplane and the nearest training points from each class. This focus on the margin is a form of geometric regularization. In the hard-margin linear case it amounts to keeping \(\|\beta\|\) small, which controls a complexity measure of the classifier (its VC dimension, or a Rademacher bound). That is a reason to expect better generalization to new data; it is not a guarantee on any given sample. The same construction highlights the support vectors: the subset of training points that actually define the decision boundary.


The flexibility of SVMs extends far beyond linear settings. Through kernel functions, the data can be treated as if they had been mapped into a higher-dimensional feature space, so that a linear rule in that space becomes a nonlinear rule in the original \(x\). The mapping itself need not be computed. In this way, SVMs combine geometric intuition, optimization, and statistical regularization into a single framework.