CtrlK
Hinge loss is a loss function used for training classifiers in machine learning. It is most closely associated with support vector machines (SVMs) and maximum-margin classification. The function penalizes predictions that are on the wrong side of the decision boundary as well as correct predictions that fall within a defined margin, producing sparse solutions where only the training examples near the boundary (called support vectors) contribute to the model.
For a binary classification problem where the true label is y in {+1, -1} and the raw model output (or decision function score) is f(x), the hinge loss is defined as:
L(y, f(x)) = max(0, 1 - y * f(x))
The quantity y * f(x) is called the functional margin. When this product is large and positive, the classifier is making a confident correct prediction and the loss is zero. When the margin is less than 1, the loss grows linearly. Specifically:
| Condition | Meaning | Loss value |
|---|---|---|
| y * f(x) >= 1 | Correct prediction with sufficient margin | 0 |
| 0 < y * f(x) < 1 | Correct prediction but inside the margin | 1 - y * f(x) |
| y * f(x) <= 0 | Misclassification | 1 - y * f(x) (at least 1) |
In a linear classifier, the decision function takes the form f(x) = w . x + b, where w is the weight vector and b is the bias term. The loss is zero only when the data point is classified correctly and lies outside the margin boundary.
Imagine you are drawing a line on a piece of paper to separate red dots from blue dots. Hinge loss is like a teacher who checks your work. If a dot is on the correct side of the line and far enough away from it, the teacher says "great, no penalty." If a dot is on the correct side but too close to the line, the teacher gives you a small penalty and says "move it farther away." If a dot is on the wrong side entirely, the teacher gives you a bigger penalty. The goal is to draw the line so that you get zero penalty, meaning all the dots are on their correct side and comfortably far from the line.
Hinge loss is the foundational loss function behind SVMs. The standard SVM optimization problem for a linear classifier can be written as:
minimize (1/2) ||w||^2 + C * sum_i max(0, 1 - y_i * (w . x_i + b))
This objective has two parts:
The hyperparameter C controls the trade-off between maximizing the margin (keeping ||w|| small) and minimizing training errors. A large C puts more emphasis on reducing training errors, while a small C favors a wider margin even if some training examples are misclassified.
The geometric margin of a linear classifier is 2/||w||. By minimizing ||w||^2 subject to the constraint that all points have a functional margin of at least 1, the SVM finds the hyperplane with the widest possible separation between the two classes. The hinge loss relaxes this hard constraint into a soft penalty, allowing some points to violate the margin at a cost proportional to their margin violation. This is known as the soft-margin SVM, introduced by Corinna Cortes and Vladimir Vapnik in 1995.
A key property of the hinge loss is that it produces sparse solutions. Any training point with a functional margin greater than or equal to 1 contributes zero loss and therefore has no influence on the learned decision boundary. Only points with a margin less than 1 (the support vectors) affect the solution. This sparsity is a practical advantage: at prediction time, the decision boundary depends only on the support vectors rather than on the entire training set.
Hinge loss has several important mathematical and practical properties:
| Property | Description |
|---|---|
| Convexity | Hinge loss is a convex function, guaranteeing that optimization will find a global minimum (in combination with a convex regularizer). |
| Piecewise linearity | The function is linear for margin values below 1 and flat (zero) for margin values at or above 1. |
| Non-differentiability | The function is not differentiable at the point y * f(x) = 1 (the "hinge" point), which requires the use of subgradient methods for optimization. |
| Non-probabilistic | Hinge loss does not produce calibrated probability estimates. It optimizes for correct classification with margin, not for estimating P(y |
| Upper bound on 0-1 loss | Hinge loss is the tightest convex upper bound on the 0-1 misclassification loss, making it a principled surrogate for the intractable 0-1 loss. |
| Robustness to outliers | Because the loss grows linearly (not quadratically) for misclassified points, it is less sensitive to extreme outliers than squared error losses. |
Because hinge loss is not differentiable at the hinge point (y * f(x) = 1), standard gradient descent cannot be applied directly. Instead, optimization relies on subgradient methods.
A subgradient generalizes the concept of a gradient to non-smooth convex functions. For the hinge loss with a linear model f(x) = w . x, the subgradient with respect to the weight vector w is:
In practice, stochastic subgradient descent works well for training SVMs with hinge loss. At each step, the algorithm picks a training example, computes the subgradient of the combined regularization and hinge loss terms, and updates the weights. This approach is the basis of the Pegasos algorithm (Shalev-Shwartz et al., 2007), which provides efficient online SVM training.
The squared hinge loss is a smooth variant defined as:
L_squared(y, f(x)) = max(0, 1 - y * f(x))^2
This variant squares the hinge loss value, making the function differentiable everywhere (including at the hinge point). Key differences from the standard hinge loss include:
| Aspect | Hinge loss | Squared hinge loss |
|---|---|---|
| Formula | max(0, 1 - yf(x)) | [max(0, 1 - yf(x))]^2 |
| Differentiability | Not differentiable at yf(x) = 1 | Differentiable everywhere |
| Penalty growth | Linear for margin violations | Quadratic for margin violations |
| Outlier sensitivity | More robust | More sensitive to large violations |
| Sparsity | Sparser support vectors | More (but smaller) non-zero support vectors |
| Optimization | Requires subgradient methods | Compatible with standard gradient descent |
The squared hinge loss is available in scikit-learn's LinearSVC (via loss='squared_hinge', which is the default) and is sometimes preferred when smooth optimization is desired.
The binary hinge loss can be extended to multi-class problems. Two prominent formulations exist:
Proposed by Crammer and Singer (2001), this defines the multi-class hinge loss as:
L(x, t) = max(0, 1 + max_{j != t} (w_j . x) - w_t . x)
where t is the true class label, w_t is the weight vector for the correct class, and w_j are the weight vectors for all other classes. The loss is zero when the score for the correct class exceeds the highest score among all incorrect classes by at least a margin of 1.
An alternative approach sums over all incorrect classes:
L(x, t) = sum_{j != t} max(0, 1 + w_j . x - w_t . x)
This formulation penalizes every incorrect class that comes within the margin of the correct class, not just the most competitive one. It tends to produce more conservative classifiers but is computationally more expensive.
In PyTorch, the MultiMarginLoss function implements a multi-class hinge loss:
loss(x, y) = sum_i max(0, margin - x[y] + x[i])^p / x.size(0)
where p can be set to 1 (standard hinge) or 2 (squared hinge), and margin defaults to 1.
Hinge loss is one of several loss functions used for classification. The table below compares it to other common choices:
| Loss function | Formula | Probabilistic | Differentiable | Typical use case |
|---|---|---|---|---|
| Hinge loss | max(0, 1 - yf(x)) | No | No (at yf(x)=1) | SVMs, maximum-margin classifiers |
| Logistic loss | log(1 + exp(-yf(x))) | Yes | Yes | Logistic regression |
| Cross-entropy loss | -sum y_k log(p_k) | Yes | Yes | Neural networks, multi-class problems |
| Squared loss | (y - f(x))^2 | No | Yes | Regression (not ideal for classification) |
| Exponential loss | exp(-yf(x)) | No | Yes | AdaBoost |
Cross-entropy loss (log loss) and hinge loss are the two most common choices for classification tasks. The key differences are:
Logistic loss, defined as log(1 + exp(-y * f(x))), is closely related to cross-entropy for binary problems. Both logistic loss and hinge loss are convex surrogates for the 0-1 loss. Logistic loss decreases exponentially as the margin increases but never reaches zero, meaning every training point always contributes some gradient. Hinge loss reaches exactly zero for points with margin at least 1, creating the sparsity that characterizes SVMs.
scikit-learn provides hinge loss through several classifiers:
from sklearn.svm import LinearSVC
# Standard hinge loss SVM
clf = LinearSVC(loss='hinge', C=1.0)
clf.fit(X_train, y_train)
predictions = clf.predict(X_test)
The SGDClassifier also supports hinge loss and is suitable for large-scale datasets because it uses stochastic gradient descent:
from sklearn.linear_model import SGDClassifier
# SGD-based linear SVM with hinge loss (default)
clf = SGDClassifier(loss='hinge', alpha=0.0001, max_iter=1000)
clf.fit(X_train, y_train)
Additionally, sklearn.metrics.hinge_loss computes the average hinge loss for evaluating classification model predictions.
PyTorch provides several hinge-related loss functions:
torch.nn.HingeEmbeddingLoss: Measures similarity between two inputs. For label y = 1, the loss is the input value itself. For y = -1, the loss is max(0, margin - input). This is typically used in metric learning and semi-supervised learning rather than standard classification.torch.nn.MultiMarginLoss: Implements multi-class hinge loss (Crammer-Singer style). It computes the sum of max(0, margin - x[y] + x[i])^p over all incorrect classes, divided by the number of classes.torch.nn.MarginRankingLoss: Given inputs x1, x2, and label y in {+1, -1}, computes max(0, -y * (x1 - x2) + margin). This is used in ranking tasks.For standard binary SVM-style hinge loss in PyTorch, users typically implement it directly:
import torch
def hinge_loss(output, target):
return torch.mean(torch.clamp(1 - target * output, min=0))
Several smoothed versions of the hinge loss have been proposed to enable standard gradient-based optimization while preserving the margin-based behavior:
Rennie and Srebro's smoothed hinge loss:
where t = y * f(x). This variant is quadratic near the hinge point and linear for large margin violations, providing a differentiable function that closely approximates the original hinge loss.
Huberized hinge loss applies a similar idea, replacing the sharp corner at the hinge point with a quadratic segment controlled by a smoothing parameter. This approach draws from the Huber loss used in robust regression.
The development of hinge loss is inseparable from the history of support vector machines. The conceptual foundations trace back to the work of Vladimir Vapnik and Alexei Chervonenkis, who introduced the Generalized Portrait Method for pattern recognition in 1964 at the Institute of Control Sciences in Moscow. This early work established the theoretical groundwork for optimal separating hyperplanes.
In the early 1990s, Vapnik, working at Bell Labs alongside collaborators including Corinna Cortes, Bernhard Boser, and Isabelle Guyon, extended these ideas into practical algorithms. The kernel trick allowed the linear separation concept to work in high-dimensional feature spaces. In their landmark 1995 paper "Support-Vector Networks," Cortes and Vapnik introduced the soft-margin SVM formulation that explicitly uses the hinge loss to handle non-separable data. This formulation allowed the SVM to tolerate some margin violations while still seeking the maximum-margin hyperplane.
The theoretical justification for using the hinge loss comes from statistical learning theory. The hinge loss is the tightest convex upper bound on the 0-1 misclassification loss, which itself is computationally intractable to optimize directly (NP-hard in general). This property makes hinge loss a principled choice for classification.
The multi-class extension by Crammer and Singer in 2001 generalized the binary hinge loss to problems with more than two classes, and subsequent work by researchers including Lee, Lin, and Wahba explored further variations and their statistical properties.
Hinge loss is a strong choice in certain scenarios:
Hinge loss is less suitable when: