Hyperplane

A hyperplane is a flat, affine subspace of dimension n-1 embedded in an n-dimensional space, defined by the linear equation $w \cdot x + b = 0$, where w is a normal vector and b is a scalar offset ^[1][12]. It generalizes a line in two dimensions and a plane in three dimensions to spaces of arbitrary dimensionality, and every hyperplane divides the space into two half-spaces. In machine learning and statistics, hyperplanes serve as decision boundaries that separate data into distinct classes, which makes them foundational to linear classifiers such as support vector machines, logistic regression, and perceptrons ^[3][5].

ELI5: explain like I'm 5

Imagine you have a pile of red balls and a pile of blue balls on a table. You want to draw a line with a ruler so that all the red balls are on one side and all the blue balls are on the other. That line is a hyperplane in two dimensions.

Now imagine the balls are floating in the air instead of sitting on a flat table. You would need a flat sheet of paper (a plane) to separate the red ones from the blue ones. That sheet of paper is a hyperplane in three dimensions.

If you had even more dimensions (which is hard to picture, but computers handle it just fine), the separator would still be a flat surface one dimension thinner than the space it sits in. That flat separator is always called a hyperplane. The goal of many machine learning algorithms is to find the best position for this separator so that it divides the data as cleanly as possible.

What is a hyperplane in mathematics?

Formally, a hyperplane in the Euclidean space $\mathbb{R}^n$ is the set of all points x satisfying a linear equation:

$$a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = b$$

where $a = (a_1, a_2, \ldots, a_n)$ is a nonzero normal vector perpendicular to the hyperplane and b is a scalar offset (also called the bias). Using vector notation, this can be written more compactly as:

$$w \cdot x + b = 0$$

where w is the weight (normal) vector and b is the bias term ^[1][12]. Every hyperplane divides the ambient space into two half-spaces: the set of points where $w \cdot x + b > 0$ and the set where $w \cdot x + b < 0$.

Key properties

• A hyperplane in $\mathbb{R}^n$ has dimension $n-1$ (equivalently, codimension 1) ^[12].
• Any hyperplane that passes through the origin is a linear subspace; otherwise it is an affine subspace.
• A hyperplane has exactly two unit normal vectors that point in opposite directions.
• The signed distance from a point $x_0$ to the hyperplane $w \cdot x + b = 0$ is $(w \cdot x_0 + b) / \lVert w \rVert$, where $\lVert w \rVert$ is the Euclidean norm of $w$ ^[1][7]. The sign of this quantity tells you which half-space the point lies in, and its absolute value gives the perpendicular distance to the hyperplane.

Dimensional examples

The following table shows what a hyperplane looks like in spaces of increasing dimension.

Scroll sideways for more →

What are the types of hyperplanes?

Linear (vector) hyperplanes

A linear hyperplane passes through the origin and is defined by $w \cdot x = 0$. It is a true linear subspace of $\mathbb{R}^n$.

Affine hyperplanes

An affine hyperplane does not necessarily pass through the origin. It is defined by $w \cdot x = b$ where $b$ is not zero. An affine hyperplane can be seen as a translated copy of a linear hyperplane. Most hyperplanes encountered in machine learning are affine because the bias term b shifts the decision boundary away from the origin to better fit the data.

Projective hyperplanes

In projective geometry, a projective hyperplane is a subspace of codimension 1 in a projective space. Unlike Euclidean hyperplanes, it takes two projective hyperplanes to separate points in a projective space.

How is a hyperplane used in classification?

In supervised classification, a decision boundary is the surface that separates the regions of feature space assigned to different classes. When this surface is a hyperplane, the classifier is called a linear classifier. The model assigns a class label to an input $x$ based on which side of the hyperplane it falls on:

• If $w \cdot x + b > 0$, predict class +1
• If $w \cdot x + b < 0$, predict class -1

Classifying a new point therefore reduces to evaluating the sign of $w \cdot x + b$, a single dot product and comparison ^[3][7]. Several widely used algorithms produce hyperplane decision boundaries, as summarized below.

Scroll sideways for more →

What is the maximum-margin hyperplane?

Support vector machines provide the most direct and well-studied application of hyperplanes in classification. The core idea is to find the hyperplane that maximizes the margin, the distance between the hyperplane and the nearest training points from each class. This maximum-margin hyperplane is the unique separating hyperplane that lies halfway between the two classes and is as far as possible from the closest point of either class ^[3][13].

The approach traces to Corinna Cortes and Vladimir Vapnik, whose 1995 paper "Support-Vector Networks" introduced the modern formulation. They described the method this way: "The support-vector network is a new learning machine for two-group classification problems" in which "input vectors are non-linearly mapped to a very high-dimension feature space" where "a linear decision surface is constructed" with "special properties" that "ensure high generalization ability of the learning machine" ^[3]. The earlier algorithm of Bernhard Boser, Isabelle Guyon, and Vapnik (1992) framed the same goal directly: "A training algorithm that maximizes the margin between the training patterns and the decision boundary is presented" ^[4].

Geometric margin

Given a separating hyperplane w . x + b = 0, the geometric margin is the perpendicular distance from the hyperplane to the closest data point of either class. For a correctly classified point $(x_i, y_i)$ with $y_i \in \{+1, -1\}$, the functional margin is $y_i(w \cdot x_i + b)$. The geometric margin normalizes this by the magnitude of w:

$$\gamma = \frac{y_i(w \cdot x_i + b)}{\lVert w \rVert}$$

The overall margin of the classifier is the minimum geometric margin across all training points ^[3][10].

Hard-margin SVM

When the training data is perfectly linearly separable, the hard-margin SVM solves the following optimization problem:

Minimize $\frac{1}{2}\lVert w \rVert^2$

Subject to $y_i(w \cdot x_i + b) \ge 1$ for all $i = 1, \ldots, m$

Minimizing $\lVert w \rVert^2$ is equivalent to maximizing the margin $2 / \lVert w \rVert$. The constraint ensures that every training point lies on the correct side of the margin boundary ^[3][8].

Soft-margin SVM

Real-world data is rarely perfectly separable. Corinna Cortes and Vladimir Vapnik introduced the soft-margin formulation in 1995, which allows some points to violate the margin by introducing slack variables $\xi_i \ge 0$ ^[3]:

Minimize $\frac{1}{2}\lVert w \rVert^2 + C \sum_i \xi_i$

Subject to $y_i(w \cdot x_i + b) \ge 1 - \xi_i$ and $\xi_i \ge 0$

The regularization parameter $C$ controls the trade-off between maximizing the margin and minimizing classification errors. A large C penalizes misclassifications heavily (narrower margin), while a small C allows more violations (wider margin).

Support vectors

The training points that lie exactly on the margin boundaries (where $y_i(w \cdot x_i + b) = 1$) are called support vectors. These are the only data points that influence the position and orientation of the optimal hyperplane ^[3][10]. If a non-support-vector point were removed from the training set or moved (as long as it stays outside the margin), the resulting hyperplane would remain unchanged. In practice the number of support vectors is often a small fraction of the training set, which is why SVMs can be both compact and accurate.

Dual formulation

The SVM optimization problem can be reformulated using Lagrange multipliers $\alpha_i$ into its dual form:

Maximize $\sum_i \alpha_i - \frac{1}{2} \sum_i \sum_j \alpha_i \alpha_j y_i y_j (x_i \cdot x_j)$

Subject to $\sum_i \alpha_i y_i = 0$ and $0 \le \alpha_i \le C$

The weight vector is recovered as $w = \sum_i \alpha_i y_i x_i$. Points with $\alpha_i > 0$ are the support vectors. The dual form is significant because the data appears only through dot products $x_i \cdot x_j$, which enables the kernel trick ^[3][4].

How does the kernel trick relate to hyperplanes?

Many real-world datasets are not linearly separable in the original input space. The kernel method addresses this by implicitly mapping data into a higher-dimensional feature space where a linear hyperplane can separate the classes. Bernhard Boser, Isabelle Guyon, and Vladimir Vapnik proposed this approach in 1992 ^[4].

How it works

A mapping function $\phi$ transforms each input $x$ into a higher-dimensional space, $x \to \phi(x)$. In this transformed space, a linear hyperplane can separate data that was nonlinearly distributed in the original space. The kernel function $K(x_i, x_j) = \phi(x_i) \cdot \phi(x_j)$ computes the dot product in the higher-dimensional space without explicitly performing the transformation. This avoids the computational cost of working in a potentially infinite-dimensional space ^[4][9].

In the dual SVM formulation, every occurrence of the dot product $x_i \cdot x_j$ is simply replaced with $K(x_i, x_j)$.

Common kernel functions

Scroll sideways for more →

The RBF kernel is the most widely used nonlinear kernel in practice because it can model complex decision boundaries while having only one hyperparameter ($\gamma$) to tune ^[9].

Interpretation in feature space

The resulting decision boundary in the original input space is nonlinear (it can be a curve, a closed region, or any complex shape), but in the high-dimensional feature space it remains a flat hyperplane. The kernel trick therefore lets SVMs find nonlinear decision boundaries while retaining the mathematical elegance and optimization guarantees of linear hyperplane separation ^[4][9].

How does the perceptron learn a hyperplane?

The perceptron, conceived by Frank Rosenblatt in 1957 and described in a 1958 paper, was one of the earliest algorithms to learn a hyperplane decision boundary ^[5]. A single-layer perceptron computes:

$$f(x) = \operatorname{sign}(w \cdot x + b)$$

The perceptron learning algorithm updates the weights each time it encounters a misclassified training example:

$$w \leftarrow w + \eta \, y_i \, x_i$$ $$b \leftarrow b + \eta \, y_i$$

where $\eta$ is the learning rate. The Perceptron Convergence Theorem guarantees that if the training data is linearly separable, the algorithm will find a separating hyperplane in a finite number of iterations ^[2][5]. However, the perceptron provides no guarantee about the quality of the hyperplane it finds; unlike SVMs, it does not maximize the margin.

Multi-layer perceptrons (the building blocks of deep learning) compose multiple layers of linear transformations and nonlinear activation functions. Each neuron in a hidden layer defines a hyperplane, and the network as a whole learns to combine many such hyperplanes to form complex, nonlinear decision boundaries.

How does logistic regression define its hyperplane?

Logistic regression is a linear classifier that models the probability that an input belongs to the positive class using the sigmoid function:

$$P(y = 1 \mid x) = \sigma(w \cdot x + b) = \frac{1}{1 + \exp(-(w \cdot x + b))}$$

The decision boundary is the hyperplane where the predicted probability equals 0.5, which occurs when w . x + b = 0. Points on the positive side of the hyperplane receive predicted probabilities greater than 0.5 and are assigned to the positive class; points on the negative side are assigned to the negative class.

The decision threshold can be adjusted from 0.5 to change the trade-off between precision and recall. Changing the threshold effectively moves the decision boundary parallel to the original hyperplane.

What is linear discriminant analysis?

Linear discriminant analysis (LDA), developed by Ronald Fisher in 1936, finds a hyperplane that separates classes by maximizing the ratio of between-class variance to within-class variance ^[6]. The projection direction $w$ is chosen to maximize the Fisher criterion:

$$J(w) = \frac{w^\top S_B w}{w^\top S_W w}$$

where $S_B$ is the between-class scatter matrix and $S_W$ is the within-class scatter matrix. The optimal solution is $w = S_W^{-1} (\mu_1 - \mu_2)$, where $\mu_1$ and $\mu_2$ are the class means. The resulting hyperplane is perpendicular to this direction.

LDA differs from SVM in that it models the class distributions (assuming Gaussian distributions with equal covariance), while SVM makes no distributional assumptions and focuses solely on the margin ^[6][7].

How are hyperplanes used for multiclass classification?

A single hyperplane can only separate two classes. To handle problems with $K > 2$ classes, several strategies exist.

Scroll sideways for more →

For example, in a 5-class problem, OvR trains 5 binary classifiers (each learning one hyperplane), while OvO trains 10 binary classifiers.

What is the separating hyperplane theorem?

The theoretical foundation for using hyperplanes as class separators comes from the hyperplane separation theorem in convex optimization. The theorem states ^[1][14]:

If A and B are two disjoint, nonempty convex sets in R^n, then there exists a nonzero vector $v$ and a scalar $c$ such that $v \cdot x \ge c$ for all $x \in A$ and $v \cdot y \le c$ for all $y \in B$.

This is the weak separation form. A stronger version adds that if both sets are closed and at least one is compact, then strict separation is possible with a gap between two parallel hyperplanes ^[1][14].

The theorem guarantees that whenever two classes form disjoint convex sets in feature space, a separating hyperplane exists. The SVM optimization problem then finds the one with the maximum margin.

Where else do hyperplanes appear in machine learning?

Decision trees and piecewise-linear boundaries

Decision trees that split on a single feature at each node create axis-aligned hyperplanes. Each split is a hyperplane perpendicular to one feature axis. The overall decision boundary is a piecewise combination of these axis-aligned hyperplanes, forming rectangular regions in feature space. Oblique decision trees allow splits on linear combinations of features, producing arbitrarily oriented hyperplanes at each node.

Neural networks

Each neuron in a neural network computes w . x + b followed by a nonlinear activation function. Before the activation, the neuron defines a hyperplane in its input space. A single hidden layer with enough neurons can approximate any continuous decision boundary by composing many such hyperplanes. Deep neural networks create hierarchical compositions of hyperplanes across multiple layers, which enables them to learn highly complex, nonlinear boundaries.

Dimensionality reduction and random projections

Hyperplanes also play a role in dimensionality reduction. Principal component analysis (PCA) projects data onto a lower-dimensional subspace defined by hyperplanes. Random projection methods, supported by the Johnson-Lindenstrauss lemma, use random hyperplanes to project high-dimensional data into fewer dimensions while approximately preserving pairwise distances.

Clustering

Spectral clustering and some variants of k-means use hyperplanes to partition feature space. In particular, max-margin clustering extends the SVM framework to unsupervised settings by finding the hyperplane that separates unlabeled data with the largest margin.

Why does margin maximization improve generalization?

The margin is the perpendicular distance between the two parallel hyperplanes $w \cdot x + b = +1$ and $w \cdot x + b = -1$. These are called the margin boundaries. The distance between them is $2 / \lVert w \rVert$ ^[3][13].

To see why: pick any point $x_+$ on the positive margin boundary $(w \cdot x_+ + b = 1)$ and compute its distance to the negative margin boundary $(w \cdot x + b = -1)$. The distance is:

$$d = \frac{|(w \cdot x_+ + b) - (-1)|}{\lVert w \rVert} = \frac{|1 - (-1)|}{\lVert w \rVert} = \frac{2}{\lVert w \rVert}$$

Maximizing $2 / \lVert w \rVert$ is equivalent to minimizing $\lVert w \rVert^2$, which is the SVM objective. A larger margin means the classifier is more robust to small perturbations in the test data, which generally leads to better generalization ^[3][10].

When were hyperplanes first used in machine learning?

The concept of hyperplanes has roots in 19th-century mathematics and has been applied in machine learning since the mid-20th century.

Scroll sideways for more →

What practical issues affect hyperplane classifiers?

Feature scaling

The position and orientation of the optimal hyperplane depend on the scale of each feature. Features with larger numeric ranges can dominate the dot product $w \cdot x$, causing the hyperplane to be biased toward those features. Normalization or standardization of features before training is therefore important for algorithms that rely on hyperplane decision boundaries.

High-dimensional spaces

In high-dimensional settings (where the number of features $p$ is large relative to the number of samples $n$), linear classifiers can often find a separating hyperplane even when no real pattern exists in the data. This is because, in high dimensions, random point sets become increasingly easy to separate with a hyperplane (Cover's theorem) ^[11]. Regularization and cross-validation are needed to avoid overfitting in such scenarios.

Computational complexity

Training a hard-margin SVM requires solving a quadratic programming problem. The time complexity is roughly $O(n^2 p)$ to $O(n^3)$ depending on the solver, where $n$ is the number of training samples and $p$ is the number of features ^[10]. The kernel trick does not change the asymptotic complexity but affects the constant factor, since kernel evaluations replace simple dot products.

Summary of key formulas

Scroll sideways for more →

See also

• Support vector machine
• Decision boundary
• Kernel support vector machines
• Perceptron
• Linear regression
• Logistic regression
• Activation function
• Convex optimization
• Regularization
• Dimensionality reduction

References

1. Boyd, S. and Vandenberghe, L. (2004). *Convex Optimization*. Cambridge University Press. ↩
2. Vapnik, V. and Chervonenkis, A. (1964). "A note on one class of perceptrons." *Automation and Remote Control*, 25(1). ↩
3. Cortes, C. and Vapnik, V. (1995). "Support-vector networks." *Machine Learning*, 20(3), 273-297. ↩
4. Boser, B., Guyon, I., and Vapnik, V. (1992). "A training algorithm for optimal margin classifiers." *Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory*, 144-152. ↩
5. Rosenblatt, F. (1958). "The perceptron: A probabilistic model for information storage and organization in the brain." *Psychological Review*, 65(6), 386-408. ↩
6. Fisher, R.A. (1936). "The use of multiple measurements in taxonomic problems." *Annals of Eugenics*, 7(2), 179-188. ↩
7. Hastie, T., Tibshirani, R., and Friedman, J. (2009). *The Elements of Statistical Learning*, 2nd edition. Springer. ↩
8. Bishop, C.M. (2006). *Pattern Recognition and Machine Learning*. Springer. ↩
9. Scholkopf, B. and Smola, A.J. (2002). *Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond*. MIT Press. ↩
10. Burges, C.J.C. (1998). "A tutorial on support vector machines for pattern recognition." *Data Mining and Knowledge Discovery*, 2(2), 121-167. ↩
11. Cover, T.M. (1965). "Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition." *IEEE Transactions on Electronic Computers*, EC-14(3), 326-334. ↩
12. "Hyperplane." *Wikipedia*. https://en.wikipedia.org/wiki/Hyperplane ↩
13. "Support vector machine." *Wikipedia*. https://en.wikipedia.org/wiki/Support_vector_machine ↩
14. "Hyperplane separation theorem." *Wikipedia*. https://en.wikipedia.org/wiki/Hyperplane_separation_theorem ↩