Representation learning

Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand. A representation is whatever encoding of the input the model actually computes with: a vector of image statistics, a word vector, the activations of a hidden layer. The working assumption of the field is that this encoding, rather than the classifier sitting on top of it, usually decides whether a learning system succeeds.^[1]

The canonical statement of that assumption is the review by Yoshua Bengio, Aaron Courville and Pascal Vincent, first posted to arXiv in June 2012 and published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2013. Its abstract opens: "The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data."^[1][2] The paper surveys unsupervised feature learning and deep learning across probabilistic models, auto-encoders, manifold learning and deep networks, and it names "the quest for AI" as what motivates the design of more powerful representation-learning algorithms.^[1]

The term is prominent enough to name a conference. The International Conference on Learning Representations (ICLR) describes itself as "the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning"; its 2026 edition, the fourteenth, runs 23-27 April 2026 at the Riocentro Convention and Event Center in Rio de Janeiro.^[3]

From feature engineering to learned features

Before deep networks became standard, most of the effort in an applied project went into feature engineering: deciding by hand which measurements to extract from raw data before a classifier ever saw them. Computer vision accumulated a catalog of such descriptors, the best known being David Lowe's scale-invariant keypoints, published in the International Journal of Computer Vision in 2004.^[4] These descriptors encoded human intuitions about invariance: a useful feature should survive changes in scale, rotation or lighting.

The alternative is to let the network find its own features. The 2012 ImageNet convolutional neural network of Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, later known as AlexNet, reported error rates "considerably better than the previous state-of-the-art results" while learning every filter from data.^[5] Two years later, Jason Yosinski, Jeff Clune, Bengio and Hod Lipson measured how far those learned filters travel. Lower layers of an image network learn generic patterns such as Gabor filters, higher layers specialize to the original task, and "initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset."^[6] That finding turned pretrained networks into reusable assets and made transfer learning the default workflow in vision.

Yann LeCun, Bengio and Hinton put the idea at the center of their 2015 Nature review of deep learning, which they describe as methods that learn a hierarchy of representations, each layer converting the one below into something slightly more abstract.^[7]

What makes a representation useful

Bengio, Courville and Vincent frame the target as separating the explanatory factors of variation that raw data entangles, and note that although domain knowledge can help design a representation, learning with generic priors is an alternative route to the same separation. Their review also connects representation learning to density estimation and to the geometry of the data manifold, and treats the question of what objective actually produces a good representation as unresolved.^[1]

Because no agreed definition of "good" exists, representations are judged empirically. Several evaluation protocols became standard, and a method's ranking can change depending on which one is used.

Scroll sideways for more →

Linear probes are popular because they isolate what the frozen representation already contains, but a representation that probes well does not always fine-tune well. The masked autoencoder authors reported that "linear probing and fine-tuning results are largely uncorrelated," and noted that linear probing is also poorly correlated with transfer performance on tasks such as object detection.^[18]

Learning paradigms

Generative and reconstruction-based methods

The earliest deep representation learners were reconstruction models. Hinton and Ruslan Salakhutdinov's 2006 Science paper showed that "high-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors," and that a properly initialized deep autoencoder beat principal component analysis at dimensionality reduction.^[8] The initialization trick came from the deep belief network of Hinton, Simon Osindero and Yee Whye Teh, published the same year in Neural Computation, which stacked Boltzmann machines and trained them one layer at a time.^[9] Bengio, Pascal Lamblin, Dan Popovici and Hugo Larochelle generalized the recipe to continuous inputs and tested empirically whether greedy layer-wise unsupervised pre-training actually helps the optimization.^[10]

Pascal Vincent, Larochelle, Bengio and Pierre-Antoine Manzagol added corruption to the input in 2008, producing the denoising autoencoder: forcing a model to reconstruct a clean signal from a damaged copy makes the intermediate code carry structure rather than an identity mapping.^[11] The variational autoencoder of Diederik Kingma and Max Welling, posted in December 2013, gave this family an explicit probabilistic latent space by reparameterizing a variational lower bound so that it could be optimized with ordinary stochastic gradient methods.^[40]

Contrastive self-supervision

Contrastive learning trains a representation by pulling together encodings of related inputs and pushing apart unrelated ones. Aaron van den Oord, Yazhe Li and Oriol Vinyals set out the modern version in 2018 with contrastive predictive coding, which predicts future observations in latent space using "a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples," tested on speech, images, text and reinforcement learning in 3D environments.^[12]

The 2019-2021 wave that followed made self-supervised learning competitive with supervised pre-training on images.

Scroll sideways for more →

The DINO paper also reported that self-supervised vision transformer features contain explicit semantic segmentation information that supervised transformers and convolutional networks do not show.^[16]

Masked prediction

The other dominant self-supervised recipe hides part of the input and asks the model to fill it in. BERT, released by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova in October 2018, pre-trains "deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers," then fine-tunes with minimal architecture changes. It reported a GLUE score of 80.5%, MultiNLI accuracy of 86.7%, SQuAD v1.1 test F1 of 93.2 and SQuAD v2.0 test F1 of 83.1, and set state of the art on eleven natural language processing tasks.^[17] The masked language model objective became the standard pre-training task for text encoders.

The vision equivalent took three more years. The masked autoencoder of He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar and Ross Girshick masks 75% of image patches and reconstructs the missing pixels with an asymmetric encoder-decoder, which both accelerates training by three times or more and improves accuracy; a plain ViT-Huge reached 87.8% on ImageNet-1K.^[18] In speech, wav2vec 2.0 masked the input in latent space and solved a contrastive task over quantized latents, reaching 1.8/3.3 word error rate on the clean and other Librispeech test sets with the full labeled set, and 4.8/8.2 using ten minutes of labeled data with 53,000 hours of unlabeled pre-training.^[19]

Language supervision

A third route supervises images with the text that happens to accompany them. OpenAI's CLIP, trained on 400 million image-text pairs collected from the internet, learns by predicting which caption goes with which image. Benchmarked on over 30 vision datasets, it matched the accuracy of the original ResNet-50 on ImageNet zero-shot, without using any of the 1.28 million training examples.^[20] The result showed that a representation trained purely on paired text could be steered to new tasks through language rather than through labeled examples.

Embeddings in practice

The commercial face of representation learning is the embedding: a fixed-length vector standing in for a word, sentence, image or user, on which similarity search and downstream models operate. Tomas Mikolov, Kai Chen, Greg Corrado and Jeff Dean's 2013 word2vec paper made this concrete for text, learning word embeddings from 1.6 billion words in under a day and improving on earlier neural approaches at a fraction of the compute.^[21] Contextual encoders such as BERT later replaced fixed per-word vectors with representations that vary by sentence.^[17]

Evaluation of text embeddings consolidated around shared benchmarks. MTEB, introduced in 2022 by Niklas Muennighoff, Nouamane Tazi, Loic Magne and Nils Reimers, argued that embeddings were being judged on too narrow a slice of tasks, and benchmarked 33 models on 8 tasks across 58 datasets and 112 languages; no single method dominated.^[22] Its 2025 successor MMTEB expanded this to more than 500 quality-controlled tasks across over 250 languages, adding instruction following, long-document retrieval and code retrieval, and introduced a downsampling method based on inter-task correlation to keep the cost manageable. In that evaluation, the best-performing publicly available model was the 560-million-parameter multilingual-e5-large-instruct, ahead of language models with billions of parameters, though those larger models did top particular language subsets and task categories.^[23] Google's Gemini Embedding, described in a March 2025 paper, reported state-of-the-art results across the multilingual, English and code portions of MMTEB.^[24]

These vectors are what vector databases, semantic search and retrieval-augmented generation operate on, generally by cosine similarity between embeddings.

Foundation models as representation learners

The 2021 Stanford report that popularized the term foundation model defined the category as models "that are trained on broad data at scale and are adaptable to a wide range of downstream tasks," naming BERT, DALL-E and GPT-3 as examples, and noted that scale produces emergent capabilities while encouraging homogenization across applications.^[25] Read against the 2013 review, a foundation model is a representation learner whose representation is reused so widely that its defects propagate downstream.

In vision, Meta's DINOv2 pushed the frozen-feature idea to scale, arguing that self-supervised pre-training yields all-purpose features when trained on enough curated data, and training a 1-billion-parameter vision transformer that was then distilled into smaller models exceeding OpenCLIP on most benchmarks.^[26] Its successor DINOv3, announced on 14 August 2025 with the paper posted the previous day, was trained with 7 billion parameters on 1.7 billion images and introduced Gram anchoring to stop dense features degrading during long training runs. The paper reports that a single frozen backbone outperforms the specialized state of the art across a broad range of settings without fine-tuning, and Meta reported that a variant trained on satellite and aerial imagery cut the average error in measuring tree canopy height in a region of Kenya from 4.1 meters to 1.2 meters compared with DINOv2.^[27][28]

The same logic reaches robotics and biology. V-JEPA 2, announced by Meta on 11 June 2025, is a 1.2-billion-parameter joint embedding predictive architecture pre-trained on more than a million hours of video plus a million images, then trained further on 62 hours of robot data; Meta reported 65% to 80% success rates on pick-and-place tasks with unfamiliar objects in new and unseen environments.^[29] In protein science, Alexander Rives and colleagues showed in 2021 that scaling unsupervised learning to 250 million protein sequences produces representations in which biological structure and function emerge without supervision, the basis for later protein language models.^[30]

Disentanglement, geometry and interpretability

Disentangling the explanatory factors behind the data is the goal the 2013 review sets out, and a line of work took it literally, trying to build models whose individual latent coordinates track individual factors.^[1] The beta-VAE of Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess and colleagues, presented at ICLR 2017, pursued this with a constrained variational framework built on the variational autoencoder.^[31]

That line of work ran into a hard result. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ratsch, Sylvain Gelly, Bernhard Scholkopf and Olivier Bachem showed theoretically in 2019 that unsupervised learning of disentangled representations is impossible without inductive biases on both models and data, then trained more than 12,000 models across seven datasets to check the practical consequences. Methods did enforce the properties they were built to enforce, but picking a well-disentangled model required supervision, and greater disentanglement did not reduce the sample complexity of downstream learning.^[32]

Attention shifted toward describing the geometry that models actually learn. Kiho Park, Yo Joong Choe and Victor Veitch formalized the linear representation hypothesis, "the idea that high-level concepts are represented linearly as directions in some representation space," connecting it to linear probing and to steering model behavior, and validating it on LLaMA-2.^[33] Sparse autoencoders took the same premise into interpretability: Anthropic reported in May 2024 that dictionary learning applied to the middle layer of Claude 3 Sonnet extracted millions of features corresponding to concepts such as cities, people and programming syntax.^[34]

A more speculative claim is the Platonic representation hypothesis of Minyoung Huh, Brian Cheung, Tongzhou Wang and Phillip Isola, who argue that representations in deep networks are converging across time, domains and modalities, with vision and language models increasingly measuring distances between data points in similar ways as they scale. The authors present it as a hypothesis and discuss counterexamples themselves.^[38]

Limitations

Learned features are not automatically the right features. Robert Geirhos and co-authors define shortcuts as "decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios," and argue in Nature Machine Intelligence that a large share of deep learning's known failures share this cause.^[35] A representation that scores well on a held-out split can be encoding the dataset's incidental regularities rather than the intended concept.

The value of pre-training is also narrower than early transfer results suggested. He, Girshick and Dollar found that ImageNet pre-training "speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy," and reached 50.9 AP on COCO object detection training from random initialization with no external data.^[36]

Representations inherit whatever their training corpus contains. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama and Adam Kalai showed in 2016 that word embeddings trained on Google News encoded gender stereotypes geometrically, as a direction in the vector space, and that gender-neutral words were linearly separable from gender-definitional ones; their debiasing method removes stereotypical associations while keeping legitimate ones.^[37] Because a single foundation model now supplies representations to many applications, such defects are inherited rather than isolated, which is the homogenization risk the 2021 Stanford report named.^[25] This is one of the mechanisms behind algorithmic bias in deployed systems.

Finally, the field's evaluations remain contested. The MTEB authors concluded that no single embedding method wins across tasks and that the field has yet to converge on a universal text embedding,^[22] while Locatello and co-authors reported that increased disentanglement did not reduce the sample complexity of downstream learning in their sweep, which weakens the main practical argument for pursuing it.^[32]

Developments in 2025 and 2026

Recent work has concentrated on scale, on modalities beyond text and images, and on theory. DINOv3 (August 2025) and V-JEPA 2 (June 2025) both push frozen general-purpose backbones into settings previously handled by task-specific models.^[27][29] On the text side, MMTEB was accepted at ICLR, and its finding that a 560-million-parameter model was the strongest publicly available entrant cuts against the assumption that embedding quality tracks parameter count.^[23] Gemini Embedding, meanwhile, pursued one model covering many languages plus code.^[24]

Theory has partly caught up with practice. In a paper posted in July 2026, William Yang, Andrew Saxe and Peter Latham derived exact solutions for how linear concept directions emerge during training in linear networks, finding that data geometry and target geometry jointly determine the final representation, that depth improves abstraction, and that initialization scale caps how much abstraction is achievable; they also report an attenuation law by which nonlinearities weaken abstraction in activations relative to preactivations.^[39] Representation learning remains the organizing theme of ICLR, whose fourteenth meeting is scheduled for Rio de Janeiro in April 2026.^[3]

See also

• Self-supervised learning
• Embeddings
• Latent space
• Transfer learning
• Deep learning
• Foundation models

References

1. ^Yoshua Bengio, Aaron Courville, Pascal Vincent, "Representation Learning: A Review and New Perspectives," arXiv:1206.5538, submitted 24 June 2012, revised 23 April 2014. arxiv.org/...1206.5538
2. ^dblp record for Bengio, Courville and Vincent, "Representation Learning: A Review and New Perspectives," IEEE Transactions on Pattern Analysis and Machine Intelligence 35(8):1798-1828, 2013, doi 10.1109/TPAMI.2013.50. dblp.org/...BengioCV13
3. ^International Conference on Learning Representations, official site (ICLR 2026, 23-27 April 2026, Riocentro Convention and Event Center, Rio de Janeiro). iclr.cc
4. ^dblp record for David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision 60(2):91-110, 2004. dblp.org/...Lowe04
5. ^Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Advances in Neural Information Processing Systems 25, 2012. papers.nips.cc/...b9d6b76c8436e924a68c45b-Abstract
6. ^Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, "How transferable are features in deep neural networks?," arXiv:1411.1792, 6 November 2014 (NIPS 2014). arxiv.org/...1411.1792
7. ^dblp record for Yann LeCun, Yoshua Bengio, Geoffrey E. Hinton, "Deep learning," Nature 521(7553):436-444, 2015, doi 10.1038/nature14539. dblp.org/...LeCunBH15
8. ^G. E. Hinton and R. R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," Science, vol. 313, p. 504, 28 July 2006. cs.toronto.edu/...science.pdf
9. ^dblp record for Geoffrey E. Hinton, Simon Osindero, Yee Whye Teh, "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation 18(7):1527-1554, 2006. dblp.org/...HintonOT06
10. ^Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, "Greedy Layer-Wise Training of Deep Networks," Advances in Neural Information Processing Systems 19, 2006. papers.nips.cc/...0c067105aeb2fae32403405-Abstract
11. ^dblp record for Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol, "Extracting and composing robust features with denoising autoencoders," ICML 2008, pp. 1096-1103. dblp.org/...VincentLBM08
12. ^Aaron van den Oord, Yazhe Li, Oriol Vinyals, "Representation Learning with Contrastive Predictive Coding," arXiv:1807.03748, 10 July 2018. arxiv.org/...1807.03748
13. ^Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick, "Momentum Contrast for Unsupervised Visual Representation Learning," arXiv:1911.05722, 13 November 2019 (CVPR 2020). arxiv.org/...1911.05722
14. ^Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton, "A Simple Framework for Contrastive Learning of Visual Representations," arXiv:2002.05709, 13 February 2020. arxiv.org/...2002.05709
15. ^Jean-Bastien Grill et al., "Bootstrap your own latent: A new approach to self-supervised Learning," arXiv:2006.07733, 13 June 2020. arxiv.org/...2006.07733
16. ^Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, Armand Joulin, "Emerging Properties in Self-Supervised Vision Transformers," arXiv:2104.14294, 29 April 2021. arxiv.org/...2104.14294
17. ^Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," arXiv:1810.04805, 11 October 2018. arxiv.org/...1810.04805
18. ^Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, Ross Girshick, "Masked Autoencoders Are Scalable Vision Learners," arXiv:2111.06377, 11 November 2021. arxiv.org/...2111.06377
19. ^Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," arXiv:2006.11477, 20 June 2020. arxiv.org/...2006.11477
20. ^Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh et al., "Learning Transferable Visual Models From Natural Language Supervision," arXiv:2103.00020, 26 February 2021. arxiv.org/...2103.00020
21. ^Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, "Efficient Estimation of Word Representations in Vector Space," arXiv:1301.3781, 16 January 2013. arxiv.org/...1301.3781
22. ^Niklas Muennighoff, Nouamane Tazi, Loic Magne, Nils Reimers, "MTEB: Massive Text Embedding Benchmark," arXiv:2210.07316, 13 October 2022. arxiv.org/...2210.07316
23. ^Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Marton Kardos et al., "MMTEB: Massive Multilingual Text Embedding Benchmark," arXiv:2502.13595, 19 February 2025, accepted at ICLR. arxiv.org/...2502.13595
24. ^Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer et al., "Gemini Embedding: Generalizable Embeddings from Gemini," arXiv:2503.07891, 10 March 2025. arxiv.org/...2503.07891
25. ^Rishi Bommasani et al. (Stanford Center for Research on Foundation Models), "On the Opportunities and Risks of Foundation Models," arXiv:2108.07258, 16 August 2021. arxiv.org/...2108.07258
26. ^Maxime Oquab, Timothee Darcet, Theo Moutakanni et al., "DINOv2: Learning Robust Visual Features without Supervision," arXiv:2304.07193, 14 April 2023. arxiv.org/...2304.07193
27. ^Oriane Simeoni, Huy V. Vo, Maximilian Seitzer et al., "DINOv3," arXiv:2508.10104, 13 August 2025. arxiv.org/...2508.10104
28. ^Meta AI, "DINOv3: self-supervised vision model," 14 August 2025. ai.meta.com/...dinov3-self-supervised-vision-model
29. ^Meta AI, "Introducing V-JEPA 2," 11 June 2025. ai.meta.com/...v-jepa-2-world-model-benchmarks
30. ^Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, Rob Fergus, "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences," Proceedings of the National Academy of Sciences 118(15):e2016239118, 2021. doi.org/...pnas.2016239118
31. ^dblp record for Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, Alexander Lerchner, "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework," ICLR 2017. dblp.org/...HigginsMPBGBML17
32. ^Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ratsch, Sylvain Gelly, Bernhard Scholkopf, Olivier Bachem, "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations," Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4114-4124, 2019. proceedings.mlr.press/...locatello19a
33. ^Kiho Park, Yo Joong Choe, Victor Veitch, "The Linear Representation Hypothesis and the Geometry of Large Language Models," arXiv:2311.03658, 7 November 2023 (ICML 2024). arxiv.org/...2311.03658
34. ^Anthropic, "Mapping the Mind of a Large Language Model," 21 May 2024. anthropic.com/...mapping-mind-language-model
35. ^Robert Geirhos, Jorn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, Felix A. Wichmann, "Shortcut Learning in Deep Neural Networks," arXiv:2004.07780, 16 April 2020; published in Nature Machine Intelligence, doi 10.1038/s42256-020-00257-z. arxiv.org/...2004.07780
36. ^Kaiming He, Ross Girshick, Piotr Dollar, "Rethinking ImageNet Pre-training," arXiv:1811.08883, 21 November 2018. arxiv.org/...1811.08883
37. ^Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai, "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings," arXiv:1607.06520, 21 July 2016. arxiv.org/...1607.06520
38. ^Minyoung Huh, Brian Cheung, Tongzhou Wang, Phillip Isola, "The Platonic Representation Hypothesis," arXiv:2405.07987, 13 May 2024. arxiv.org/...2405.07987
39. ^William W. Yang, Andrew M. Saxe, Peter E. Latham, "How are linear representations learned? Exact solutions to the dynamics of abstraction," arXiv:2607.08843, 9 July 2026. arxiv.org/...2607.08843
40. ^Diederik P. Kingma, Max Welling, "Auto-Encoding Variational Bayes," arXiv:1312.6114, 20 December 2013. arxiv.org/...1312.6114