# Dimensionality reduction > Source: https://aiwiki.ai/wiki/dimensionality_reduction > Updated: 2026-06-21 > Categories: Data Science, Machine Learning > From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution. **Dimensionality reduction** is the process of transforming data from a high-dimensional space into a lower-dimensional representation that retains as much of the meaningful structure of the original data as possible. It is a foundational technique in [machine learning](/wiki/machine_learning), statistics, signal processing, and data visualization, used to combat the [curse of dimensionality](/wiki/curse_of_dimensionality), reveal hidden structure, compress data, and accelerate downstream learning algorithms. The field spans more than a century: the oldest method, [principal component analysis](/wiki/principal_component_analysis) (PCA), was introduced by Karl Pearson in 1901, while the most widely used modern visualization methods, [t-SNE](/wiki/t_sne) (2008) and [UMAP](/wiki/umap) (2018), are now standard figures in essentially every quantitative discipline.[1][27][34] A landmark 2006 result by Geoffrey Hinton and Ruslan Salakhutdinov showed that deep [autoencoder](/wiki/autoencoder) networks can learn low-dimensional codes that work "much better than principal components analysis as a tool to reduce the dimensionality of data," a finding often cited as a catalyst of the modern deep learning era.[24] ## Definition Formally, dimensionality reduction is a mapping $f: \mathbb{R}^D \rightarrow \mathbb{R}^d$ (with $d \ll D$) applied to a dataset $X = \{x_1, x_2, \dots, x_n\} \subset \mathbb{R}^D$, producing a new dataset $Y = \{y_1, y_2, \dots, y_n\} \subset \mathbb{R}^d$. The goal is for $Y$ to preserve certain properties of $X$, such as variance, pairwise distances, local neighborhoods, class separability, or probability distributions. Depending on the method, the mapping $f$ may be linear or nonlinear, parametric or non-parametric, supervised or unsupervised, and invertible (reconstructive) or not. Dimensionality reduction methods are broadly divided into two families: 1. **Feature selection**: choosing a subset of the original features, discarding the rest. See [feature selection](/wiki/feature_selection). 2. **Feature extraction**: constructing new features as functions of the originals. See [feature extraction](/wiki/feature_extraction). Both families aim to reduce $D$ while preserving predictive or descriptive information. ## ELI5 (Explain Like I'm 5) Imagine you have a huge spreadsheet with 10,000 columns describing each of your friends: their height, favorite color, birthday, shoe size, how loudly they laugh, and thousands more things. It is way too much to look at. Dimensionality reduction is like squishing that spreadsheet down to just two or three columns that still let you tell your friends apart. It throws away the repeated or boring stuff and keeps the interesting differences, sort of like making a cartoon of a face: you only need a few strokes for the eyes, nose, and mouth, and the cartoon still looks like the person. ## Why is dimensionality reduction needed? ### The curse of dimensionality The phrase *curse of dimensionality*, coined by Richard E. Bellman in 1961 while studying dynamic programming, refers to the collection of phenomena that make high-dimensional spaces behave in counterintuitive ways.[6] As dimensionality grows, the volume of the space grows exponentially, so a fixed-size sample of data becomes vanishingly sparse. Several concrete problems follow: - **Distance concentration**: in high dimensions under many common distributions, the ratio between the distances to the nearest and farthest neighbors of a point approaches one, so nearest-neighbor searches lose discriminative power. - **Sample complexity**: the amount of data required to reliably estimate densities, decision boundaries, or smooth functions grows exponentially in the number of dimensions. - **Hubness**: in high-dimensional k-nearest-neighbor graphs, a few points (hubs) appear in the neighbor lists of many other points, skewing algorithms that rely on neighborhood structure. - **Overfitting**: models with many input features easily memorize idiosyncratic noise, harming generalization. Dimensionality reduction mitigates these problems by projecting data into a space whose dimension is closer to its *intrinsic* dimensionality, the number of degrees of freedom that actually generate the variation observed in the data. ### Visualization Humans perceive at most three spatial dimensions. To inspect, cluster, or sanity-check large datasets visually, practitioners routinely project data to two or three dimensions using methods such as [t-SNE](/wiki/t_sne), [UMAP](/wiki/umap), or the first two principal components. Good visualization embeddings let researchers spot clusters, outliers, trajectories, and mislabeled points that would be impossible to notice in the raw data. ### Computational efficiency Many learning algorithms scale poorly with input dimension. K-nearest neighbors, kernel methods, Gaussian processes, and dense neural networks all become slower and more memory-hungry as $D$ grows. Reducing the dimensionality to a compact code accelerates training and inference, and sometimes improves accuracy by removing noise and redundancy. ### Noise reduction and denoising High-dimensional data often lies near a lower-dimensional manifold, with deviations from that manifold corresponding to noise. Projecting to the manifold, or to its principal directions of variation, tends to filter out this noise. This is why methods like [principal component analysis](/wiki/principal_component_analysis) and [autoencoders](/wiki/autoencoder) are common preprocessing steps in image, audio, and sensor applications. ### Storage and compression A 2048-pixel grayscale image lives in a 2048-dimensional vector space, but only a small manifold of that space contains realistic images. Dimensionality reduction exploits this redundancy. Lossy compression schemes based on discrete cosine transforms (JPEG), wavelets, and learned autoencoders are all forms of dimensionality reduction applied to media. ### Multicollinearity and interpretability Real-world features are often correlated. Dimensionality reduction combines correlated features into independent or less-correlated components, stabilizing regression coefficients and making downstream models more interpretable when the reduced components themselves have meaningful interpretations. ## Taxonomy of methods There is no single axis along which to classify dimensionality reduction methods, but the following table summarizes the main algorithms and a few key properties. *Supervised* indicates whether label information is used, *Linear* indicates whether the mapping is linear in the input features, and *Global/Local* indicates which type of structure the method is designed to preserve. | Method | Year | Key reference | Linear | Supervised | Structure preserved | |---|---|---|---|---|---| | Principal Component Analysis | 1901 | Pearson | Yes | No | Global variance | | Factor Analysis | 1904 | Spearman | Yes | No | Latent common factors | | Multidimensional Scaling (classical) | 1938 | Young and Householder | Yes | No | Global pairwise distances | | Linear Discriminant Analysis | 1936 | Fisher | Yes | Yes | Class separability | | Independent Component Analysis | 1994 | Comon | Yes | No | Statistical independence | | Sammon Mapping | 1969 | Sammon | No | No | Global distances (weighted) | | Kernel PCA | 1998 | Scholkopf, Smola, Muller | No | No | Nonlinear variance | | Isomap | 2000 | Tenenbaum, de Silva, Langford | No | No | Geodesic distances | | Locally Linear Embedding | 2000 | Roweis, Saul | No | No | Local linearity | | Laplacian Eigenmaps | 2001 | Belkin, Niyogi | No | No | Local neighborhoods | | Diffusion Maps | 2005 | Coifman, Lafon | No | No | Random-walk geometry | | Random Projection | 1984 | Johnson, Lindenstrauss | Yes | No | Pairwise distances (approx) | | Probabilistic PCA | 1999 | Tipping, Bishop | Yes | No | Gaussian latent variable | | Deep Autoencoder | 2006 | Hinton, Salakhutdinov | No | No | Reconstructive code | | t-SNE | 2008 | van der Maaten, Hinton | No | No | Local neighborhoods | | UMAP | 2018 | McInnes, Healy, Melville | No | No | Local and some global | Orthogonal to this taxonomy, one can also distinguish: - **Parametric** methods (e.g., PCA, LDA, autoencoders) produce an explicit function that can embed new points after training. - **Non-parametric** methods (e.g., classical t-SNE, Isomap, LLE) only embed the specific training points; new points require an extra out-of-sample extension. - **Reconstructive** methods (e.g., PCA, autoencoders) can map reduced codes back to the original space approximately. - **Non-reconstructive** methods (e.g., t-SNE, UMAP) produce an embedding without an inverse mapping. ## Linear methods ### Principal Component Analysis [Principal component analysis](/wiki/principal_component_analysis_pca) (PCA) is the oldest and most widely used dimensionality reduction technique. It was introduced by Karl Pearson in his 1901 paper *On Lines and Planes of Closest Fit to Systems of Points in Space*, and developed independently by Harold Hotelling in 1933.[1][3] Pearson formulated the method geometrically, as the problem of finding the best-fitting low-dimensional affine subspace to a cloud of points.[1] Hotelling reformulated it algebraically in terms of eigenvectors of the covariance matrix.[3] Given centered data $X \in \mathbb{R}^{n \times D}$ (each column has zero mean), the sample covariance matrix is $$\Sigma = \frac{1}{n-1} X^\top X.$$ PCA computes the eigendecomposition $\Sigma = V \Lambda V^\top$, where the columns of $V$ are orthonormal eigenvectors (the *principal components* or *loading vectors*) and $\Lambda$ is a diagonal matrix of nonnegative eigenvalues sorted in decreasing order. The projection of a data point $x$ onto the top $d$ components is $$y = V_d^\top x,$$ where $V_d$ contains the first $d$ columns of $V$. Equivalently, PCA can be computed from the [singular value decomposition](/wiki/singular_value_decomposition) $X = U S V^\top$, which is numerically preferable because it avoids forming $\Sigma$ explicitly. PCA has three equivalent interpretations: 1. **Variance maximization**: the first principal component is the unit vector $v_1$ that maximizes $\mathrm{Var}(X v_1)$; subsequent components maximize variance subject to being orthogonal to the previous ones. 2. **Reconstruction error minimization**: PCA finds the orthogonal projection of rank $d$ that minimizes the squared reconstruction error $\|X - \hat{X}\|_F^2$. This is the classical Eckart-Young theorem. 3. **Decorrelation**: the principal components are the directions in which the data are uncorrelated. The fraction of variance captured by the first $d$ components is $\sum_{i=1}^d \lambda_i / \sum_{i=1}^D \lambda_i$, which gives an intuitive rule for picking $d$ (e.g., retain enough components to explain 95% of variance). **Strengths**: simple, fast, optimal under the Frobenius norm, closed-form solution, parametric so new points can be projected trivially, invertible up to the truncation. **Limitations**: purely linear, sensitive to feature scaling (features with larger units dominate), sensitive to outliers (because it uses squared error), cannot capture nonlinear manifold structure such as a Swiss roll. ### Singular Value Decomposition and Truncated SVD The singular value decomposition factorizes any real matrix $X$ as $X = U S V^\top$, where $U$ and $V$ are orthogonal and $S$ is a diagonal matrix of nonnegative singular values in decreasing order. Truncated SVD keeps only the top $d$ singular triplets, yielding the best rank-$d$ approximation of $X$ in both the Frobenius and spectral norms (Eckart-Young-Mirsky theorem). Truncated SVD is closely related to PCA but does not require centering, which makes it practical for sparse matrices (centering would destroy sparsity). In natural language processing it is the core computation behind [latent semantic analysis](/wiki/latent_semantic_analysis), which applies truncated SVD to a term-document matrix to obtain low-dimensional document and term embeddings.[9] ### Linear Discriminant Analysis [Linear Discriminant Analysis](/wiki/linear_discriminant_analysis) (LDA), introduced by Ronald A. Fisher in his 1936 paper *The Use of Multiple Measurements in Taxonomic Problems*, is a supervised linear method.[4] Given labeled data with $C$ classes, LDA seeks directions that maximize between-class variance relative to within-class variance. Formally, LDA maximizes the Rayleigh quotient $$J(w) = \frac{w^\top S_B w}{w^\top S_W w},$$ where $S_B$ is the between-class scatter and $S_W$ is the within-class scatter. The optimal $w$ vectors are the generalized eigenvectors of $S_W^{-1} S_B$. LDA can extract at most $C - 1$ components and is primarily used as a supervised preprocessing step for classification tasks. ### How does LDA differ from PCA? Whereas PCA seeks directions that explain variance in the data as a whole, LDA seeks directions that separate the classes. PCA is unsupervised and ignores labels; LDA is supervised and uses them. The two can give very different projections on the same dataset. ### Independent Component Analysis [Independent Component Analysis](/wiki/independent_component_analysis) (ICA), formalized by Pierre Comon in 1994, seeks a linear transformation such that the resulting components are as statistically independent as possible, not merely uncorrelated.[11] ICA assumes the observed signals are linear mixtures of independent, non-Gaussian source signals and attempts to invert the mixing. A classic application is the *cocktail party problem*, separating individual voices from microphone recordings. Algorithms such as FastICA and Infomax estimate the unmixing matrix by maximizing non-Gaussianity measures like negentropy or kurtosis. ICA is useful as dimensionality reduction when interpretable source signals (rather than variance-capturing components) are desired, for example in EEG, fMRI, and audio processing. ### Factor Analysis Factor analysis, originating with Charles Spearman's 1904 work on intelligence testing, assumes that observed variables are linear combinations of a smaller number of latent common factors plus independent noise:[2] $$x = W z + \mu + \epsilon, \quad z \sim \mathcal{N}(0, I), \quad \epsilon \sim \mathcal{N}(0, \Psi),$$ where $\Psi$ is diagonal. The loadings $W$ are estimated by maximum likelihood or the method of principal factors, often followed by rotation (varimax, promax) to improve interpretability. Unlike PCA, factor analysis is not rotation-invariant and permits unique noise on each feature, which makes it more appropriate when observation noise differs across features. ### Random Projection and the Johnson-Lindenstrauss Lemma [Random projection](/wiki/random_projection) is a surprisingly powerful family of methods in which the projection matrix is drawn at random rather than learned from data. Its theoretical foundation is the [Johnson-Lindenstrauss lemma](/wiki/johnson_lindenstrauss_lemma), proved in 1984 by William B. Johnson and Joram Lindenstrauss, which states:[8] For any $0 < \varepsilon < 1$ and any set $X$ of $n$ points in $\mathbb{R}^D$, there exists a linear map $f : \mathbb{R}^D \to \mathbb{R}^m$ with $m = O(\varepsilon^{-2} \log n)$ such that for all $u, v \in X$, $$(1 - \varepsilon) \|u - v\|^2 \le \|f(u) - f(v)\|^2 \le (1 + \varepsilon) \|u - v\|^2.$$ The target dimension depends only logarithmically on the number of points and not at all on the original dimension. Moreover, the classical proof is constructive: with high probability, a matrix whose entries are i.i.d. Gaussian (or Rademacher) random variables, suitably scaled, realizes the map. Practical implementations include Gaussian random projection and the sparse random projection of Achlioptas. Random projection is attractive when $D$ is extremely large, for streaming or one-pass settings, and as a preprocessing step before other dimensionality reduction methods. It is data-oblivious, which is both its strength (no training) and its weakness (it does not adapt to the data's structure). ### Multidimensional Scaling Multidimensional scaling (MDS), in its classical form introduced by Young and Householder in 1938 and popularized by Torgerson and Gower, takes as input a matrix of pairwise dissimilarities between $n$ objects and produces coordinates in $\mathbb{R}^d$ such that the Euclidean distances approximate the input dissimilarities.[5] Classical (metric) MDS solves this via an eigendecomposition of the doubly centered squared-distance matrix, and in the special case where the dissimilarities are Euclidean distances it is equivalent to PCA. Non-metric MDS (Shepard, Kruskal) preserves only the *rank order* of the dissimilarities and is solved iteratively. MDS is widely used in psychometrics, ecology, and marketing research where only similarity judgments are available. ### Probabilistic PCA Probabilistic PCA (PPCA), introduced by Tipping and Bishop in 1999, embeds PCA in a probabilistic latent variable framework:[15] $$x = Wz + \mu + \epsilon, \quad z \sim \mathcal{N}(0, I), \quad \epsilon \sim \mathcal{N}(0, \sigma^2 I).$$ Maximum likelihood estimation recovers the same principal subspace as classical PCA, but PPCA offers additional advantages: it handles missing data naturally through the EM algorithm, it provides a proper likelihood for model comparison and mixture modeling, and it generalizes smoothly to factor analysis (by replacing $\sigma^2 I$ with a diagonal $\Psi$) and to Bayesian PCA. ## Nonlinear and manifold methods Linear methods are insufficient when data lies on a curved manifold embedded in high-dimensional space. The canonical toy example is the *Swiss roll*: a two-dimensional sheet rolled into three dimensions. PCA will flatten the roll and confuse points that are far apart along the sheet but close together in the ambient space. Nonlinear or *manifold learning* methods aim to recover the underlying low-dimensional coordinates while respecting the curvature of the manifold. See [manifold learning](/wiki/manifold_learning) for an overview. ### Kernel PCA [Kernel PCA](/wiki/kernel_pca), introduced by Bernhard Scholkopf, Alexander Smola, and Klaus-Robert Muller in a 1998 paper in *Neural Computation*, generalizes PCA via the kernel trick.[14] Instead of computing the eigendecomposition of the covariance matrix in input space, kernel PCA computes the eigendecomposition of the $n \times n$ centered kernel (Gram) matrix $K$ with entries $K_{ij} = k(x_i, x_j)$, where $k$ is a positive semidefinite kernel such as the Gaussian (RBF) kernel $k(x, y) = \exp(-\|x - y\|^2 / 2\sigma^2)$ or a polynomial kernel. This corresponds to performing standard PCA in a (possibly infinite-dimensional) feature space induced by the kernel, without ever explicitly computing the feature map. Kernel PCA can capture nonlinear structure, but it scales as $O(n^2)$ in memory and $O(n^3)$ in time for the eigendecomposition, and out-of-sample extension requires re-applying the kernel. ### Isomap Isomap (isometric feature mapping), introduced by Joshua Tenenbaum, Vin de Silva, and John Langford in a 2000 *Science* paper, extends classical MDS by replacing Euclidean distances with *geodesic* distances measured along the data manifold.[17] The algorithm has three steps: 1. **Build a neighborhood graph** connecting each point to its $k$ nearest neighbors (or to all points within a radius $\epsilon$). 2. **Compute geodesic distances** as shortest-path distances on this graph (Dijkstra or Floyd-Warshall). 3. **Apply classical MDS** to the resulting geodesic distance matrix to produce a low-dimensional embedding. Isomap recovers the true intrinsic geometry of a manifold when the data is sufficiently dense, converges to the true geodesic distances in the limit of infinite data, and on the Swiss roll unrolls the sheet into a flat rectangle. Its weaknesses are sensitivity to the choice of $k$ or $\epsilon$ (too large causes *short-circuiting* across folds of the manifold), high computational cost on large datasets, and the requirement that the manifold be isometric to a convex subset of Euclidean space. ### Locally Linear Embedding [Locally linear embedding](/wiki/locally_linear_embedding) (LLE), introduced by Sam Roweis and Lawrence Saul in the same 2000 issue of *Science* as Isomap, takes a local-to-global approach.[16] The intuition is that a smooth manifold looks locally linear, so each point can be approximated as a linear combination of its neighbors, and the same linear weights should describe the point in the low-dimensional embedding. The algorithm has three steps: 1. **Find neighbors**: identify the $k$ nearest neighbors of each point. 2. **Compute reconstruction weights**: for each point $x_i$, find weights $w_{ij}$ that minimize $\|x_i - \sum_j w_{ij} x_j\|^2$ subject to $\sum_j w_{ij} = 1$, using only the neighbors of $x_i$. 3. **Compute the embedding**: find low-dimensional coordinates $y_i$ that minimize $\sum_i \|y_i - \sum_j w_{ij} y_j\|^2$. This reduces to a sparse eigenvalue problem. LLE is appealing because its optimization has no local minima and it captures the local geometry faithfully, but it can distort global structure and is sensitive to the neighborhood parameter and to noise. Variants include Hessian LLE, Modified LLE, and Local Tangent Space Alignment (LTSA). ### Laplacian Eigenmaps Laplacian eigenmaps, introduced by Mikhail Belkin and Partha Niyogi (conference version 2001, journal version 2003 in *Neural Computation*), embed the data by solving a generalized eigenvalue problem on the graph Laplacian of a neighborhood graph.[20] The algorithm constructs a weighted adjacency matrix $W$ (often using a Gaussian heat kernel on neighbor pairs), forms the graph Laplacian $L = D - W$ where $D$ is the diagonal degree matrix, and computes the eigenvectors corresponding to the smallest nonzero eigenvalues of the generalized problem $L y = \lambda D y$. These eigenvectors are the low-dimensional coordinates. The method is motivated by the observation that on a Riemannian manifold, the Laplace-Beltrami operator governs heat diffusion and its eigenfunctions provide a natural basis of smooth, locality-preserving functions.[20] Laplacian eigenmaps therefore produce embeddings in which neighboring points in the graph are close in the embedding, which is a natural criterion for clustering-friendly projections. Laplacian eigenmaps is closely connected to spectral clustering and to normalized cuts. ### Hessian Eigenmaps and Diffusion Maps Hessian eigenmaps (Donoho and Grimes, 2003) replaces the Laplacian with a Hessian-based operator and can recover the manifold's coordinate chart even when the manifold is not convex, removing one of Isomap's limitations.[21] Diffusion maps (Coifman and Lafon, 2005) view the data as a Markov chain and embed points so that Euclidean distances in the embedding approximate the diffusion distance (distance between probability distributions after several random-walk steps).[23] The diffusion distance averages over all paths between two points and is therefore robust to noise and short-circuiting. ### Sammon Mapping Sammon mapping (John Sammon, 1969) is an early nonlinear MDS variant that minimizes a weighted stress function[7] $$E = \frac{1}{\sum_{i