Foundation Models

A foundation model is a machine learning model trained on broad data, generally with self-supervision at scale, that can be adapted to a wide range of downstream tasks. The term was introduced in a 2021 Stanford research report to describe a role in an AI development pipeline, not one architecture, company, product, or minimum parameter count.^[1] A foundation model can support many applications after prompting, retrieval, or additional training, while a deployed artificial intelligence system can also include data pipelines, tools, access controls, interfaces, and human procedures.

Breadth and adaptability are the central ideas. A model trained for one fixed task is not a foundation model merely because it is large, and a model does not have to generate text to qualify. Language encoders, autoregressive generators, and models that connect images with text can all play the foundation-model role. The same reuse that makes broad pretraining useful also creates shared dependencies: improvements and defects can affect many downstream systems. The original report called this combination of leverage and inherited risk "homogenization."^[1]

Foundation models overlap with large language models, generative AI, and multimodal models, but the terms are not synonyms. They describe different properties: language modelling concerns data and objectives, generative AI concerns producing new instances, multimodality concerns input or output types, and foundation models concern broad pretraining plus downstream adaptation.

Definition and scope

The original definition has three connected elements: broad training data, training generally based on self-supervised learning at scale, and adaptation to many downstream tasks.^[1] "Generally" matters. Self-supervision is common because targets can be derived from the data itself, but the definition does not require one universal objective. Nor does it set a fixed threshold for parameters, training tokens, money, or computing operations.

"Broad" is relative to the intended domain. A text model can be broad across subjects, genres, and tasks without accepting images. A scientific model can be broad within a research domain without being a general conversational system. Evidence that a model is a foundation model therefore comes from its training scope and demonstrated adaptability, not from a marketing label or a single benchmark score.

The model should also be distinguished from the system built around it. Stanford researchers later clarified that "foundation" describes the model's role and does not claim that the model alone is the foundation of AI. A foundation model remains one component of a larger system.^[2] Retrieval indexes, external tools, moderation rules, user interfaces, and monitoring can change system behaviour without changing the underlying model.

Scroll sideways for more →

Historical development

Broad pretraining and reuse predate the phrase "foundation model." They grew from transfer learning, representation learning, and improvements in deep learning systems. The 2017 Transformer showed that sequence transduction could be performed with attention-based encoder and decoder layers rather than recurrence or convolution in the original architecture.^[3] Transformers later became common in foundation models, but the architecture is neither the definition nor a requirement for every modality.

BERT, published in 2018 and presented at NAACL in 2019, pretrained bidirectional text representations using unlabeled text and then fine-tuned the same model for multiple language-understanding tasks.^[4] GPT-3, reported in 2020, scaled autoregressive language modelling to 175 billion parameters and studied zero-shot, one-shot, and few-shot task performance through text prompts without gradient updates at use time. Its authors also reported tasks on which the model struggled and methodological problems caused by large web corpora.^[5]

CLIP, published in 2021, extended broad pretraining across images and natural-language descriptions. Its contrastive objective learned which caption matched an image, after which natural-language class descriptions could be used for zero-shot transfer to image-classification datasets.^[6] BERT, GPT-3, and CLIP were among the examples used in the Stanford report to show that the emerging pattern was wider than one task or modality.^[1]

In August 2021, Rishi Bommasani and more than one hundred collaborators published "On the Opportunities and Risks of Foundation Models." The report named the pattern and surveyed its technical foundations, applications, and societal consequences.^[1] The name was intentionally about a model's central but incomplete role. It did not claim that these systems were technically new in every respect: the report situated them within established deep learning and transfer-learning methods.

Training and adaptation

Foundation-model development usually separates broad pre-training from later adaptation. During pretraining, a model learns parameters from a large and heterogeneous collection. The target can be the next token, a masked element, a denoised representation, or the correspondence between modalities. The objective determines what statistical signal is available, while architecture, data composition, optimization, and compute constrain what the model can learn.

Scale is multidimensional. Parameter count, data volume and quality, training compute, context length, modality coverage, and post-training can all matter. A larger value on one dimension does not guarantee better performance on a specific task. Published numbers also may not be comparable when developers count data, operations, or model components differently. Claims about capability should therefore be tied to an identified model, evaluation protocol, and version.

Pretraining yields a base checkpoint, not a finished application. Downstream work can alter weights, add small trainable components, condition a frozen model, or combine it with external resources:

Scroll sideways for more →

These methods address different goals. Fine-tuning can specialize a domain representation, preference training can change response behaviour, and retrieval can provide updateable evidence without placing that evidence in model weights. None independently guarantees truth, safety, or suitability for a consequential use.

Modalities and applications

Foundation models first drew broad attention in language, but the concept is modality-neutral. A text encoder can provide reusable representations for classification or retrieval. An autoregressive text model can generate continuations and perform tasks specified in a prompt. A vision-language model can align image and text representations. Other research applies broad pretraining and adaptation to audio, video, robotics, biology, and scientific data.^[1]

Modality labels do not by themselves establish broad capability. CLIP is evidence for reusable image-text representation learning under a particular training and evaluation setup, not proof that every vision-language model is a foundation model or that CLIP solves every visual task.^[6] Similarly, GPT-3's in-context results showed task transfer under specified prompts, while the same paper documented uneven performance and data-contamination concerns.^[5]

Downstream uses range from search, classification, transcription, and content generation to software assistance and scientific modelling. The foundation model is often only the starting point. A medical or legal application, for example, needs domain evidence, validation against its intended population, access controls, and professional oversight. Broad pretraining cannot substitute for use-specific evaluation.

Transfer, emergence, and homogenization

Transfer is the practical reason to build a foundation model: knowledge encoded during broad pretraining can be reused for tasks with less task-specific data or training than starting from randomly initialized parameters. Transfer is empirical rather than automatic. It can be positive, negligible, or harmful depending on the source data, target domain, adaptation method, and evaluation.

Researchers have also studied emergent abilities, defined in one influential paper as abilities absent in smaller models but present in larger ones.^[11] Apparent discontinuities require caution. A 2023 NeurIPS paper showed that nonlinear or discontinuous metrics can make smooth performance changes appear sudden, and argued that some reported emergence is a measurement artifact.^[12] The existence of improved performance at greater scale should not be converted into a general claim that every capability appears unpredictably or that scaling alone explains it.

Homogenization concerns repeated reuse of a small set of base models. It can concentrate research and engineering effort: a correction to a base model, evaluation method, or serving system may benefit multiple applications. It also creates correlated failure. Biases, security weaknesses, licensing constraints, outages, or undocumented training choices can propagate to downstream systems that share the same dependency.^[1] A risk assessment should therefore trace both upstream inputs and downstream integrations rather than evaluate an application as if it were independent.

Evaluation and documentation

No single benchmark establishes that a foundation model is generally capable, safe, or fit for deployment. Evaluation must identify the model version, task, population, prompt or adaptation method, metric, baseline, and uncertainty. Scores across different prompting rules or data versions may not be directly comparable. Public test sets can also appear in broad training corpora, which complicates interpretation of held-out performance.^[5][14]

HELM, introduced as a holistic evaluation framework for language models, organized evaluations by scenarios, adaptations, and metrics. Its original study measured accuracy alongside calibration, robustness, fairness, bias, toxicity, and efficiency, and explicitly documented gaps in coverage.^[13] The framework illustrates why a leaderboard position is not a complete model assessment. Different deployments place different weights on error types, latency, language coverage, accessibility, privacy, and safety.

Evaluation should occur at several levels:

• Base-model evaluation tests capabilities and limitations under controlled conditions.
• Adapted-model evaluation tests the effects of fine-tuning, prompts, retrieval, or tools.
• System evaluation includes interfaces, human behaviour, access rules, and failure recovery.
• Deployment monitoring looks for distribution shift, abuse, incidents, and impacts that a static test did not capture.

Documentation makes these results interpretable. A model card can record intended uses, evaluation procedures, subgroup results, limitations, and relevant ethical considerations.^[17] Training-data documentation should describe sources, filtering, dates, licences where known, and known exclusions. In a study of the C4 web corpus, researchers found unexpected source domains, machine-generated material, benchmark examples, and filtering effects that disproportionately removed text associated with some minority identities.^[14] This demonstrates why a dataset name or size is not enough to characterize its contents.

Transparency remains incomplete. The 2025 Foundation Model Transparency Index evaluated developers across one hundred indicators covering upstream resources, model properties, and downstream use. Its authors reported persistent gaps in disclosures about training data and basic model information.^[18] The index is an assessment framework, not proof that a particular model is accurate or safe, and its scores should be read with its methodology and date.

Risks and limitations

Foundation-model risks differ by modality, access method, adaptation, and use. They should not be treated as inevitable properties of every model, but broad reuse can increase the number of systems exposed to a shared weakness.

Data quality and representation. Web-scale and institutional data can contain errors, duplicated material, stereotypes, personal information, and uneven coverage across languages or groups. Filtering can remove harmful content while introducing its own selection effects.^[14] Data audits, subgroup evaluation, and documentation address different questions and should be used together.

Memorization and privacy. Models can memorize training records. Researchers demonstrated extraction of verbatim training sequences from a language model, including information that appeared only rarely in its corpus.^[15] Separate experiments extracted training images from diffusion models.^[16] These results do not mean that every output copies a training example, but they disprove the blanket claim that learned parameters cannot reveal training data. Deduplication, access control, privacy testing, and differential privacy can mitigate different parts of the problem, each with tradeoffs.

Incorrect or misleading output. Generative foundation models can produce fluent statements unsupported by evidence, commonly described as hallucination or confabulation. NIST treats confabulation as one risk in generative-AI systems and emphasizes that risk depends on context and the larger system.^[26] Non-generative models have different error modes, such as misclassification or poor retrieval, so the term should not be generalized to every foundation model.

Security and misuse. Broadly capable models can be incorporated into phishing, impersonation, malware, or influence workflows, while also supporting defensive analysis. Risk depends on capability, access, scale, and surrounding controls. Evaluation can include red teaming, but an adversarial test samples a threat space and cannot prove that no exploitable behaviour remains.

Shared dependencies and control. Organizations that rely on an external model can be affected by changes to its weights, interface, licence, price, availability, or moderation rules. Open deployment can reduce some dependencies while creating responsibilities for local security, monitoring, and updates. Homogenization also means a single undocumented defect can affect many products.^[1]

Compute and environmental impact. Pretraining and repeated inference can require specialized hardware and substantial electricity. Reported energy and emissions depend on hardware, utilization, experiment count, data-centre efficiency, and the electricity supply. Earlier empirical work showed that development and tuning choices can dominate the footprint of a single reported training run.^[27] Comparisons should use disclosed system boundaries and measured workloads rather than extrapolating a universal figure from parameter count.

Mitigation is a lifecycle process. Data governance, evaluation, access control, incident response, human oversight, and AI alignment research address different failure pathways. A safeguard validated on one model version or application should not be assumed to transfer unchanged to another.

Openness and access

Access to a foundation model can be provided through a hosted interface, an API, downloadable parameters, or a more complete release. These choices expose different information and grant different rights:

Scroll sideways for more →

The Open Source Initiative's definition distinguishes model parameters from the broader materials needed to study and modify an AI system. It requires data information, code, and parameters under specified conditions, so a release should not be called open source solely because weights can be downloaded.^[19] Other frameworks may use different terminology. Precise descriptions such as "API-access," "open-weight," and "training code released" are more informative than a single open-versus-closed label.

Governance

The European Union's legal category is "general-purpose AI model" (GPAI model), not "foundation model." The concepts overlap, but the legal definition controls whether the EU AI Act applies. Article 3(63) defines a GPAI model by significant generality, competent performance across a wide range of distinct tasks, and integration into varied downstream systems or applications, with a pre-market research and prototyping exception.^[21]

Under Article 53, providers of GPAI models must maintain technical documentation, give downstream providers information about capabilities and limitations, adopt a policy for compliance with Union copyright law, and publish a sufficiently detailed summary of training content. The regulation provides limited exceptions from two documentation duties for qualifying free and open-source releases, but not from the copyright-policy and training-content-summary duties, and not for GPAI models with systemic risk.^[21]

A GPAI model can be classified as presenting systemic risk through high-impact capabilities or a Commission designation. Training compute greater than 10^25 floating-point operations creates a presumption of high-impact capabilities; it is not the definition of a foundation model, a universal safety boundary, or the only path to designation. Providers of systemic-risk models have additional duties for evaluation and adversarial testing, risk assessment and mitigation, serious-incident reporting, and cybersecurity.^[21]

The GPAI obligations entered into application on 2 August 2025. The European Commission states that its enforcement powers, including fines, apply from 2 August 2026, while providers of models already on the market before 2 August 2025 must comply by 2 August 2027.^[22] The General-Purpose AI Code of Practice is a voluntary compliance tool for the Act's transparency, copyright, and safety-and-security duties; the Commission and AI Board approved it in August 2025.^[23]

United States federal policy should be dated carefully. Executive Order 14110 created a "dual-use foundation model" reporting framework in 2023, but Executive Order 14179 revoked that order in January 2025 and directed review of actions taken under it.^[24] It is therefore inaccurate to describe the 2023 reporting provisions as a current standing executive-order regime. Separately, NIST's AI Risk Management Framework remains voluntary and provides a general process for managing AI risks.^[25] Its generative-AI profile is narrower than foundation models as a class because it addresses risks specific to generative systems.^[26]

Governance categories answer legal and policy questions, not scientific taxonomy alone. Whether a rule applies can depend on jurisdiction, market placement, provider role, release terms, and system use. Current legal claims should be checked against primary law and regulator guidance at the time of deployment.

References

1. ^Bommasani, Rishi, et al. "On the Opportunities and Risks of Foundation Models." Stanford Center for Research on Foundation Models, 2021, revised 2022. arxiv.org/...2108.07258
2. ^Stanford Center for Research on Foundation Models. "Reflections on Foundation Models." October 18, 2021. crfm.stanford.edu/...reflections
3. ^Vaswani, Ashish, et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
4. ^Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT, 2019. aclanthology.org/N19-1423
5. ^Brown, Tom B., et al. "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...18bfb8ac142f64a-Abstract
6. ^Radford, Alec, et al. "Learning Transferable Visual Models From Natural Language Supervision." Proceedings of Machine Learning Research 139, 2021. proceedings.mlr.press/...radford21a
7. ^Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." International Conference on Learning Representations, 2022. openreview.net/forum
8. ^Lester, Brian, Rami Al-Rfou, and Noah Constant. "The Power of Scale for Parameter-Efficient Prompt Tuning." EMNLP, 2021. aclanthology.org/2021.emnlp-main.243
9. ^Ouyang, Long, et al. "Training Language Models to Follow Instructions with Human Feedback." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...14f58805a001731-Abstract
10. ^Lewis, Patrick, et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...bc26945df7481e5-Abstract
11. ^Wei, Jason, et al. "Emergent Abilities of Large Language Models." Transactions on Machine Learning Research, 2022. arxiv.org/...2206.07682
12. ^Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. "Are Emergent Abilities of Large Language Models a Mirage?" Advances in Neural Information Processing Systems 36, 2023. proceedings.neurips.cc/...8bd7-Abstract-Conference
13. ^Liang, Percy, et al. "Holistic Evaluation of Language Models." Transactions on Machine Learning Research, 2023. arxiv.org/...2211.09110
14. ^Dodge, Jesse, et al. "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus." EMNLP, 2021. aclanthology.org/2021.emnlp-main.98
15. ^Carlini, Nicholas, et al. "Extracting Training Data from Large Language Models." 30th USENIX Security Symposium, 2021. usenix.org/...carlini-extracting
16. ^Carlini, Nicholas, et al. "Extracting Training Data from Diffusion Models." 32nd USENIX Security Symposium, 2023. usenix.org/...carlini
17. ^Mitchell, Margaret, et al. "Model Cards for Model Reporting." Conference on Fairness, Accountability, and Transparency, 2019. arxiv.org/...1810.03993
18. ^Wan, Alexander, et al. "Foundation Model Transparency Index." Stanford Center for Research on Foundation Models, December 2025. crfm.stanford.edu/fmti
19. ^Open Source Initiative. "The Open Source AI Definition, Version 1.0." October 2024. opensource.org/...open-source-ai-definition
20. ^United Kingdom Department for Science, Innovation and Technology. "Frontier AI: Capabilities and Risks, Discussion Paper." October 2023. gov.uk/...-capabilities-and-risks-discussion-paper
21. ^European Parliament and Council of the European Union. "Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence." Official Journal of the European Union, 2024. eur-lex.europa.eu/...eng
22. ^European Commission. "Guidelines for Providers of General-Purpose AI Models." Updated April 28, 2026. digital-strategy.ec.europa.eu/...es-gpai-providers
23. ^European Commission. "Drawing-Up a General-Purpose AI Code of Practice." Updated June 25, 2026. digital-strategy.ec.europa.eu/...ai-code-practice
24. ^President of the United States. "Executive Order 14179: Removing Barriers to American Leadership in Artificial Intelligence." Federal Register, January 31, 2025. federalregister.gov/...-in-artificial-intelligence
25. ^National Institute of Standards and Technology. "AI Risk Management Framework." Updated April 7, 2026. nist.gov/...ai-risk-management-framework
26. ^Autio, Chloe, et al. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, July 2024. nvlpubs.nist.gov/...NIST.AI.600-1.pdf
27. ^Strubell, Emma, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP." Proceedings of ACL, 2019. arxiv.org/...1906.02243