Latent Space

A latent space is the vector space a machine learning model maps its inputs into, where each input becomes a point (a latent vector or latent code) and the geometry of the space carries information the raw input format does not expose. The word "latent" is borrowed from statistics, where a latent variable is one that is never observed directly and has to be inferred from data that is. Latent Dirichlet allocation, for example, models each document as a mixture over unobserved topics ^[1]. Kingma and Welling state the assumption in its modern form: the data "are generated by some random process, involving an unobserved continuous random variable z" ^[2].

In a neural network the idea becomes geometric. An encoder assigns every image, sentence, molecule, or audio clip a coordinate vector, usually with far fewer dimensions than the input, and a decoder or a downstream head reads that vector back out. Because the encoder is trained to preserve whatever the objective cares about, points that are close in the latent space tend to correspond to inputs that are similar in the relevant sense, and directions in the space often correspond to interpretable changes. That property is what makes latent spaces useful for generation, editing, retrieval, and interpretability work.

The term overlaps heavily with embeddings and embedding space, and the literature does not draw a hard line between them. The usual pull is that "latent space" gets used where the space is a bottleneck inside a generative or compressive model, as in the autoencoder that Stable Diffusion runs its denoiser inside ^[21][22], while "embedding space" gets used where the map exists to support similarity search or transfer, as in CLIP's shared image and text representation ^[20]. The underlying object is the same in both cases: a learned map from inputs to vectors.

Origins and the manifold hypothesis

The geometric justification for compressing data into a small space is the manifold hypothesis, stated by Fefferman, Mitter and Narayanan as the claim that "high dimensional data tend to lie in the vicinity of a low dimensional manifold" ^[3]. A 512 by 512 color photograph has 786,432 pixel values, but the set of images that look like photographs occupies a vanishingly small, curved region of that space. If the region really is low dimensional, a model can parameterize it with a handful of coordinates.

The related motivation from representation learning is disentangling. Bengio, Courville and Vincent argued that machine learning performance depends heavily on how data is represented, because "different representations can entangle and hide more or less the different explanatory factors of variation behind the data" ^[4]. A good latent space, on this view, is one whose axes line up with the factors that actually generate the data.

Both ideas predate deep learning. Principal component analysis and singular value decomposition give linear latent coordinates, and latent semantic analysis applied SVD to document-term matrices for retrieval. Nonlinear dimensionality reduction methods followed: t-SNE places high-dimensional points in a two or three dimensional map for visualization ^[5], and UMAP builds a similar projection from Riemannian geometry and algebraic topology while claiming better preservation of global structure and faster run times ^[6]. Both are now standard tools for looking at the latent spaces of larger models, though neither is the latent space itself.

Autoencoders and variational autoencoders

The autoencoder is the most direct way to construct a latent space: train a network to reconstruct its own input through a narrow middle layer, and the activations of that layer become the code. Hinton and Salakhutdinov's 2006 Science paper made this work for deep networks, showing that "high-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors" ^[7]. Their 784-1000-500-250-30 network reduced MNIST digits to 30 numbers with an average squared reconstruction error of 3.00, against 8.01 for 30-dimensional logistic PCA and 13.87 for standard PCA. Trained on half of a set of 804,414 Reuters newswire stories, each represented as probabilities over the 2,000 commonest word stems, a 2000-500-250-125-2 autoencoder produced two-dimensional codes in which documents clustered by subject; a 10-dimensional code beat latent semantic analysis at document retrieval when the cosine of the angle between two codes was used to measure similarity ^[7].

The variational autoencoder adds a probabilistic structure to the same architecture. Kingma and Welling place a prior over the latent variable, in their worked example "the centered isotropic multivariate Gaussian", and train an inference network to approximate the intractable posterior for each datapoint, using a reparameterization of the variational lower bound so the whole thing can be optimized by gradient descent ^[2][8]. The practical consequence is that the latent space is dense and smooth rather than a scatter of isolated codes: sampling from the prior produces valid outputs, and the paper visualizes this by feeding a grid of latent coordinates, mapped through the inverse Gaussian CDF, into the decoder to produce continuous manifolds of MNIST digits and Frey faces ^[2].

Bowman and colleagues carried the idea into text, training an RNN-based VAE with "distributed latent representations of entire sentences" and showing that "by examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences" ^[9]. They also reported that the model presents a difficult learning problem requiring specific training techniques, and negative results when used for language modeling, an early sign that smooth text latent spaces are harder to obtain than image ones ^[9].

GAN latent spaces: walks, interpolation, and arithmetic

Generative adversarial networks sample a noise vector and push it through a generator, so the latent space exists by construction ^[10]. Radford, Metz and Chintala's DCGAN paper established the standard ways of probing it. They noted that "walking on the manifold that is learnt can usually tell us about signs of memorization (if there are sharp transitions) and about the way in which the space is hierarchically collapsed", and set out the test: if walking in the latent space "results in semantic changes to the image generations (such as objects being added and removed), we can reason that the model has learned relevant and interesting representations". Their interpolations between nine random points showed smooth transitions, with one row showing "a room without a window slowly transforming into a room with a giant window" ^[11].

The same paper introduced latent space arithmetic for images, explicitly borrowing from word vectors: it cites the result that "vector('King') - vector('Man') + vector('Woman') resulted in a vector whose nearest neighbor was the vector for Queen" and tests whether the generator's space behaves similarly ^[11]. Single samples were unreliable, and the authors state that "averaging the Z vector for three examplars showed consistent and stable generations that semantically obeyed the arithmetic"; a "turn" vector for face pose was built from four averaged samples of faces looking left versus right ^[11].

Two refinements followed. Tom White showed that "replacing linear interpolation with spherical linear interpolation prevents diverging from a model's prior distribution and produces sharper samples", because a straight line between two samples of a high-dimensional Gaussian passes through a region the prior almost never visits ^[12]. And StyleGAN split the latent space in two: an 8-layer mapping network converts the input latent Z into an intermediate space W, both 512-dimensional, and because W is not required to follow the density of the training data it "is therefore allowed to be disentangled" ^[13]. Karras, Laine and Aila introduced two metrics for this, perceptual path length and linear separability, to measure interpolation smoothness and how well latent points can be split by a hyperplane along binary image attributes ^[13]. StyleGAN2 added a path length regularizer that encourages good conditioning in the latent-to-image map and, as a side effect, makes the generator "significantly easier to invert" ^[14].

Inverting a real photograph into a GAN latent space turned out to need more room than the generator's own sampling space. Abdal, Qin and Wonka found that "embedding into W directly does not give reasonable results" and instead used an extended space W+, "a concatenation of 18 different 512-dimensional w vectors, one for each layer of the StyleGAN architecture that can receive input via AdaIn" ^[15]. Once an image is in W+, editing operations such as morphing, style transfer and expression transfer become moves in the latent space ^[15].

Word and sentence embeddings

The best-known latent space arithmetic comes from language. Mikolov, Yih and Zweig's NAACL 2013 paper described a vector offset method: to answer the analogy a:b as c:d, compute y = x_b - x_a + x_c from unit-normalized vectors and return the word whose embedding has the highest cosine similarity to y. With the induced representations, they report that "King - Man + Woman" gives a vector very close to "Queen" ^[16]. Their best model answered 39.6 percent of an 8,000-question syntactic analogy set correctly, from vectors trained on 320 million words of broadcast news with an 82,000-word vocabulary ^[16]. The word2vec papers released the same year made these word embeddings cheap enough to train at scale, the first of them reporting that "it takes less than a day to learn high quality word vectors from a 1.6 billion words data set" ^[17], the second adding subsampling of frequent words, negative sampling as an alternative to the hierarchical softmax, and a method for finding phrases ^[18].

The analogy demonstration has aged badly as evidence about what latent spaces contain. Nissim, van Noord and van der Goot point out that "in the default implementation of word2vec, gensim as well as the code from Bolukbasi et al., the input terms of the analogy query are not allowed to be returned" ^[19]. Remove that constraint from the widely repeated "man is to computer programmer as woman is to homemaker" style query and the answer often becomes the original word: their title, "Man is to Doctor as Woman is to Doctor", makes the point. Their conclusion is that analogies are "probably not the right tool" for diagnosing bias, and that the practice "exacerbated some possibly non-existing biases and perhaps hid others" ^[19]. Human bias in embeddings is real; the analogy trick is a poor instrument for measuring it.

Embedding spaces are now built for more than one modality at a time. CLIP trains an image encoder and a text encoder on 400 million image-text pairs using the task of predicting which caption goes with which image, producing a shared space in which a photograph and its description land near one another and new visual concepts can be named in natural language at inference time ^[20]. That aligned geometry is what allows text prompts to steer image models and what vector databases index for semantic search and retrieval augmented generation using cosine similarity.

Diffusion in latent space

The move that made high-resolution image generation affordable was running the diffusion model in a learned latent space instead of on pixels. Rombach and colleagues trained an autoencoder first, then trained the diffusion process on its codes, an approach they describe as "significantly reducing computational requirements compared to pixel-based DMs" while enabling "DM training on limited computational resources" ^[21]. They swept the downsampling factor f over {1, 2, 4, 8, 16, 32} and reported that f = 4 through f = 16 "strike a good balance between efficiency and perceptually faithful results", with f = 4 and f = 8 offering "the best conditions for achieving high-quality synthesis results"; both a mild KL penalty toward a standard normal and a vector-quantized variant were used as regularizers on the first-stage model ^[21]. This is the architecture behind latent diffusion systems generally.

Stable Diffusion v1 is the widely deployed instance: a "downsampling-factor 8 autoencoder" with an 860M-parameter U-Net and a 123M-parameter CLIP ViT-L/14 text encoder, pretrained at 256 by 256 and fine-tuned at 512 by 512 on LAION subsets ^[22]. The Diffusers implementation of that autoencoder defaults to four latent channels and a scaling factor of 0.18215, described as "the component-wise standard deviation of the trained latent space computed using the first batch of the training set", applied so the latents have unit variance for the diffusion model ^[23]. A 512 by 512 RGB image therefore becomes a 64 by 64 by 4 tensor, 48 times fewer values than the pixel grid.

How much the latent bottleneck costs in fidelity depends on its width. In the Stable Diffusion 3 paper, Esser and colleagues report that "increasing the number of latent channels d significantly boosts reconstruction performance" and settled on 16 channels because "the d=16 autoencoder exhibits better scaling performance in terms of sample FID" ^[24].

Scroll sideways for more →

Source: Stable Diffusion 3 autoencoder ablation ^[24]. Lower is better for FID and perceptual similarity; higher is better for SSIM and PSNR.

Latent space and interpretability

Reading a language model's internal space is harder than reading a generator's, because the interesting concepts are not the coordinate axes. Park, Choe and Veitch formalise the linear representation hypothesis, "the idea that high-level concepts are represented linearly as directions in some representation space", and connect it to linear probing and model steering ^[25]. Elhage and colleagues explain why the directions need not be neurons: networks store more sparse features than they have dimensions by packing them in superposition, which is what makes individual neurons polysemantic ^[26].

Sparse autoencoders attack that problem with dictionary learning: they fit an overcomplete basis over the model's activations and force each activation vector to be a sparse combination of its elements. Anthropic's October 2023 result decomposed a layer of 512 neurons into more than 4,000 features that separately represent things like DNA sequences, legal language, HTTP requests and Hebrew text, and human raters scored the features as more interpretable than the neurons ^[27]. The May 2024 follow-up extracted millions of features from the middle layer of Claude 3 Sonnet, including a feature that fires on the Golden Gate Bridge in English, Japanese, Chinese, Greek, Vietnamese and Russian text as well as on images of the bridge; nearby features in the same space included Alcatraz Island and the 1906 San Francisco earthquake ^[28]. Clamping that one feature to ten times its maximum activation produced Golden Gate Claude, a research demo that ran for 24 hours and steered nearly every answer back to the bridge ^[29]. Google DeepMind released Gemma Scope, an open suite of JumpReLU sparse autoencoders covering all layers and sub-layers of Gemma 2 2B and 9B and selected layers of 27B, to make this kind of work possible outside industry labs ^[30].

Circuit-level tracing has since found structure that spans languages. Anthropic's March 2025 study of Claude 3.5 Haiku reported that when the model is asked for the "opposite of small" across different languages, "the same core features for the concepts of smallness and oppositeness activate", triggering a concept of largeness that then gets translated into the language of the question, which points to a shared conceptual space upstream of the language-specific output. The same study found that the model plans rhyming words before writing the line that reaches them ^[31]. These are all claims about geometry inside the residual stream, which is to say about a latent space.

Reasoning in latent space

A newer line of work treats the latent space as the medium of computation rather than as a representation to be read. Hao and colleagues introduced Coconut (Chain of Continuous Thought), which feeds the model's last hidden state back in as the next input embedding instead of decoding it to a token; the authors argue that a continuous thought "can encode multiple alternative next steps, allowing the model to perform a breadth-first search (BFS) rather than committing prematurely to a single deterministic path as in CoT" ^[32]. Geiping and colleagues took a different route, iterating a recurrent block to unroll arbitrary depth at test time, and scaled a 3.5-billion-parameter proof-of-concept trained on 800 billion tokens to a compute load equivalent to a 50-billion-parameter model on reasoning benchmarks ^[33]. A July 2025 survey defines the category as performing "multi-step inference entirely in the model's continuous hidden state, eliminating token-level supervision", motivated by the claim that natural-language chain of thought limits the model's expressive bandwidth ^[34].

Work in the area continued into 2026. LatentMT applies the approach to machine translation, reporting a 2.6-billion-parameter model that spends additional recurrent computation inside its hidden states and reaches "performance comparable to models three to five times larger" across 32 translation directions, at lower training and inference compute than the larger non-recurrent alternatives. The same paper finds that recurrent computation improves quality at first but "saturates quickly afterwards" ^[35].

Limitations

Latent spaces are easy to over-read. Four recurring problems are worth naming.

Disentanglement is not free. Locatello and colleagues proved that "the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data", and backed it with more than 12,000 trained models across seven datasets in an ICML 2019 study ^[36]. Axes that look meaningful are usually the product of architectural choices, not discovery.

Geometric structure is not proof of mechanism. A June 2026 analysis of latent reasoning models evaluated Coconut and CODI against controls that lacked the proposed recurrence or curriculum and found that the supposedly diagnostic patterns, such as breadth-first-search-like frontiers and decodable arithmetic, "also appear in the controls and do not always causally affect behavior" ^[37]. Similar audits of sparse autoencoder features report that a large share of features clearing a geometric recovery threshold are causally inert when ablated or steered, up to 77 percent in a degraded autoencoder and about 9 percent in a well-trained one ^[38].

Famous demonstrations can be artifacts of the evaluation code. The word analogy result that convinced a generation of readers that embedding spaces encode relations as clean offsets depends on the default behavior of hiding the query terms from the answer, and drops out when that rule is lifted ^[19].

Finally, a latent space is a property of one model, not a canonical description of the data. That said, there is evidence for convergence between models. The Platonic Representation Hypothesis argues that as vision and language models grow they "measure distance between datapoints in a more and more alike way", converging toward a shared statistical model of reality ^[39]. In 2025 that argument acquired an operational demonstration: Jha and colleagues translated text embeddings between models with different architectures, parameter counts and training data without any paired data, via a "universal latent representation", and pointed out the consequence for vector database security, since an adversary holding only embedding vectors can recover attributes of the underlying documents ^[40].

See also

• Autoencoder
• Variational autoencoder
• Manifold hypothesis
• Representation learning
• Embeddings
• Latent diffusion

References

1. ^Blei, D. M., Ng, A. Y., & Jordan, M. I. "Latent Dirichlet Allocation." Journal of Machine Learning Research 3 (2003), 993-1022. jmlr.org/...blei03a
2. ^Kingma, D. P., & Welling, M. "Auto-Encoding Variational Bayes." arXiv:1312.6114, 20 December 2013. arxiv.org/...1312.6114
3. ^Fefferman, C., Mitter, S., & Narayanan, H. "Testing the Manifold Hypothesis." arXiv:1310.0425, 1 October 2013. arxiv.org/...1310.0425
4. ^Bengio, Y., Courville, A., & Vincent, P. "Representation Learning: A Review and New Perspectives." arXiv:1206.5538, 24 June 2012. arxiv.org/...1206.5538
5. ^van der Maaten, L., & Hinton, G. "Visualizing Data using t-SNE." Journal of Machine Learning Research 9 (2008), 2579-2605. jmlr.org/...vandermaaten08a
6. ^McInnes, L., Healy, J., & Melville, J. "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction." arXiv:1802.03426, 9 February 2018. arxiv.org/...1802.03426
7. ^Hinton, G. E., & Salakhutdinov, R. R. "Reducing the Dimensionality of Data with Neural Networks." Science 313 (5786), 28 July 2006, 504-507. cs.toronto.edu/...science.pdf
8. ^Kingma, D. P., & Welling, M. "An Introduction to Variational Autoencoders." arXiv:1906.02691, 6 June 2019. arxiv.org/...1906.02691
9. ^Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. "Generating Sentences from a Continuous Space." arXiv:1511.06349, 19 November 2015. arxiv.org/...1511.06349
10. ^Goodfellow, I. J., et al. "Generative Adversarial Networks." arXiv:1406.2661, 10 June 2014. arxiv.org/...1406.2661
11. ^Radford, A., Metz, L., & Chintala, S. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks." arXiv:1511.06434, 19 November 2015. arxiv.org/...1511.06434
12. ^White, T. "Sampling Generative Networks." arXiv:1609.04468, 14 September 2016. arxiv.org/...1609.04468
13. ^Karras, T., Laine, S., & Aila, T. "A Style-Based Generator Architecture for Generative Adversarial Networks." arXiv:1812.04948, 12 December 2018. arxiv.org/...1812.04948
14. ^Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. "Analyzing and Improving the Image Quality of StyleGAN." arXiv:1912.04958, 3 December 2019. arxiv.org/...1912.04958
15. ^Abdal, R., Qin, Y., & Wonka, P. "Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?" arXiv:1904.03189, 5 April 2019. arxiv.org/...1904.03189
16. ^Mikolov, T., Yih, W., & Zweig, G. "Linguistic Regularities in Continuous Space Word Representations." Proceedings of NAACL-HLT 2013, 746-751. aclanthology.org/N13-1090
17. ^Mikolov, T., Chen, K., Corrado, G., & Dean, J. "Efficient Estimation of Word Representations in Vector Space." arXiv:1301.3781, 16 January 2013. arxiv.org/...1301.3781
18. ^Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. "Distributed Representations of Words and Phrases and their Compositionality." arXiv:1310.4546, 16 October 2013. arxiv.org/...1310.4546
19. ^Nissim, M., van Noord, R., & van der Goot, R. "Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor." arXiv:1905.09866, 23 May 2019. arxiv.org/...1905.09866
20. ^Radford, A., et al. "Learning Transferable Visual Models From Natural Language Supervision." arXiv:2103.00020, 26 February 2021. arxiv.org/...2103.00020
21. ^Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. "High-Resolution Image Synthesis with Latent Diffusion Models." arXiv:2112.10752 (CVPR 2022), 20 December 2021. arxiv.org/...2112.10752
22. ^CompVis. "Stable Diffusion" repository README. GitHub. github.com/...stable-diffusion
23. ^Hugging Face. "AutoencoderKL." Diffusers documentation. huggingface.co/...autoencoderkl
24. ^Esser, P., Kulal, S., Blattmann, A., et al. "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." arXiv:2403.03206, 5 March 2024. arxiv.org/...2403.03206
25. ^Park, K., Choe, Y. J., & Veitch, V. "The Linear Representation Hypothesis and the Geometry of Large Language Models." arXiv:2311.03658, 7 November 2023. arxiv.org/...2311.03658
26. ^Elhage, N., et al. "Toy Models of Superposition." arXiv:2209.10652, 21 September 2022. arxiv.org/...2209.10652
27. ^Anthropic. "Decomposing Language Models Into Understandable Components." 5 October 2023. anthropic.com/...ls-into-understandable-components
28. ^Anthropic. "Mapping the Mind of a Large Language Model." 21 May 2024. anthropic.com/...mapping-mind-language-model
29. ^Anthropic. "Golden Gate Claude." 23 May 2024. anthropic.com/...golden-gate-claude
30. ^Lieberum, T., Rajamanoharan, S., Conmy, A., et al. "Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2." arXiv:2408.05147, 9 August 2024. arxiv.org/...2408.05147
31. ^Anthropic. "Tracing the Thoughts of a Large Language Model." 27 March 2025. anthropic.com/...tracing-thoughts-language-model
32. ^Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. "Training Large Language Models to Reason in a Continuous Latent Space." arXiv:2412.06769, 9 December 2024. arxiv.org/...2412.06769
33. ^Geiping, J., McLeish, S., Jain, N., et al. "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach." arXiv:2502.05171, 7 February 2025. arxiv.org/...2502.05171
34. ^Zhu, R.-J., Peng, T., Cheng, T., et al. "A Survey on Latent Reasoning." arXiv:2507.06203, 8 July 2025. arxiv.org/...2507.06203
35. ^Chen, W.-R., Magdy, S. M., Zhang, C., Zhu, W., Wang, Z., & Abdul-Mageed, M. "LatentMT: Machine Translation with Latent Reasoning." arXiv:2607.18618, 21 July 2026. arxiv.org/...2607.18618
36. ^Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., & Bachem, O. "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations." arXiv:1811.12359 (ICML 2019), 29 November 2018. arxiv.org/...1811.12359
37. ^Aswal, D., Palmeira Ferraz, T., Zhou, Y., & Peyrard, M. "Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models." arXiv:2606.12689, 10 June 2026. arxiv.org/...2606.12689
38. ^Bal, M. A. "From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness." arXiv:2607.12166, 13 July 2026. arxiv.org/...2607.12166
39. ^Huh, M., Cheung, B., Wang, T., & Isola, P. "The Platonic Representation Hypothesis." arXiv:2405.07987, 13 May 2024. arxiv.org/...2405.07987
40. ^Jha, R., Zhang, C., Shmatikov, V., & Morris, J. X. "Harnessing the Universal Geometry of Embeddings." arXiv:2505.12540, 18 May 2025. arxiv.org/...2505.12540