# LLaMA

> Source: https://aiwiki.ai/wiki/llama
> Updated: 2026-07-28
> Fact-checked: 2026-07-28
> Categories: Large Language Models, Meta AI, Natural Language Processing, Open Source AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "LLaMA." aiwiki.ai, 28 Jul 2026. https://aiwiki.ai/wiki/llama
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**LLaMA** is the original name of a family of [large language models](https://aiwiki.ai/wiki/large_language_model) developed by [Meta AI](https://aiwiki.ai/wiki/meta_ai). Meta introduced the first generation in February 2023 as "Large Language Model Meta AI" and changed the styling to **Llama** with Llama 2. The family includes pretrained base models, instruction-tuned models, small models intended for local devices, and multimodal models that accept images as well as text.[1][2]

Meta distributes Llama model [weights](https://aiwiki.ai/wiki/open_weights), tokenizers, model cards, and inference code. Those releases let developers run and modify a checkpoint without sending prompts to a Meta service. They do not include a complete reproduction package for training each model from raw data, and the weights are governed by generation-specific Meta licenses rather than an Open Source Initiative approved license.[6][13][14][15]

The architecture changed in stages. LLaMA, Llama 2, and Llama 3 used dense, autoregressive Transformer language models. Llama 3.2 added small text models and, separately, vision models built with an adapter on the Llama 3.1 language model. Llama 4 introduced early-fusion multimodal training and a mixture-of-experts architecture, so its total parameter count is much larger than the number of parameters used for any one token.[5][9][10][12]

## Scope and naming

"LLaMA" refers strictly to the first generation in Meta's 2023 paper and announcement. The later product name is "Llama", without the all-capital acronym. This article uses "Llama family" for the full sequence and preserves Meta's styling for each named release.

A Llama release can contain more than one kind of checkpoint:

- A pretrained or base checkpoint predicts continuations from its training objective. It is a starting point for adaptation, not a chat assistant with a stable conversation policy.
- An instruction-tuned checkpoint has additional post-training for dialogue, instruction following, preference alignment, and safety behavior. Meta called this line Llama 2-Chat in the second generation and generally used "Instruct" in later releases.
- A quantized checkpoint stores weights at lower numerical precision to reduce memory and compute requirements. Its behavior and benchmark results can differ from the full-precision checkpoint.
- A derivative model may combine a Llama checkpoint with third-party fine-tuning, quantization, retrieval, or serving code. Its behavior should not be attributed to the original Meta checkpoint without testing the derivative itself.

The family name also appears in related projects. [Code Llama](https://aiwiki.ai/wiki/code_llama), for example, was a code-specialized branch based on Llama 2. It is not a separate generation in the main sequence.[6]

## Release history

The table lists public model releases, not every research experiment, quantization, or community derivative. Context length is the maximum documented by Meta for the listed checkpoint family. It does not establish that a model will use every part of a long prompt accurately.

| Release | Public checkpoints | Documented context | Main change |
|---|---|---:|---|
| LLaMA, February 2023 | 7B, 13B, 33B, 65B base models | 2K tokens | Dense text models trained only on data that the paper described as publicly available |
| [Llama 2](https://aiwiki.ai/wiki/llama_2), July 2023 | 7B, 13B, 70B base and Chat models | 4K tokens | Longer pretraining, dialogue post-training, and a custom license allowing many commercial uses |
| [Llama 3](https://aiwiki.ai/wiki/llama_3), April 2024 | 8B and 70B base and Instruct models | 8K tokens | New 128,000-token vocabulary and grouped-query attention across released sizes |
| [Llama 3.1](https://aiwiki.ai/wiki/llama_3_1), July 2024 | 8B, 70B, 405B base and Instruct models | 128K tokens | Dense 405B model, long-context continued pretraining, eight supported languages, and revised output-use terms |
| [Llama 3.2](https://aiwiki.ai/wiki/llama_3_2), September 2024 | 1B and 3B text models | 128K tokens | Pruned and distilled models intended for constrained and on-device environments |
| [Llama 3.2 Vision](https://aiwiki.ai/wiki/llama_3_2_vision), September 2024 | 11B and 90B image-and-text models | 128K tokens | Vision encoder and cross-attention adapter attached to Llama 3.1 |
| [Llama 3.3](https://aiwiki.ai/wiki/llama_3_3), December 2024 | 70B Instruct | 128K tokens | Updated multilingual instruction model in the dense Llama 3 architecture |
| [Llama 4](https://aiwiki.ai/wiki/llama_4), April 2025 | Scout and Maverick, in base and Instruct forms | 10M for Scout; 1M for Maverick | Native image-and-text input, early fusion, and mixture-of-experts layers |

### The original LLaMA

The first LLaMA paper tested the claim that, for a fixed inference budget, a smaller model trained on more data could be preferable to a much larger model trained on fewer tokens. The released family had 6.7, 13.0, 32.5, and 65.2 billion parameters, conventionally labeled 7B, 13B, 33B, and 65B. Meta trained the two smaller models on one trillion tokens and the two larger models on 1.4 trillion tokens. The paper described a mixture of Common Crawl, C4, GitHub, Wikipedia, books, arXiv, and Stack Exchange data, all selected from publicly available sources.[1]

The paper compared the base models on zero-shot and few-shot academic benchmarks. Its often repeated result was narrower than a general claim that a 13B model "beat GPT-3": LLaMA 13B scored above GPT-3 175B on most of the reported common-sense reasoning, question-answering, reading-comprehension, and code benchmarks under the paper's evaluation settings. It did not outperform every larger model on every task. The paper also found low absolute truthfulness scores and measured social bias and toxic-generation risks.[1]

Meta did not initially publish the weights for unrestricted use. It offered case-by-case access to researchers and selected organizations under a noncommercial research license. The release therefore differed both technically and legally from the later community-license releases.[2]

### Llama 2

Llama 2 retained the main architecture of the first generation, doubled the context window from 2,048 to 4,096 tokens, and trained the public 7B, 13B, and 70B models on two trillion tokens. The paper says the pretraining corpus was a new mix of publicly available data and did not include Meta user data. The public 70B model used [grouped-query attention](https://aiwiki.ai/wiki/grouped_query_attention) to reduce the memory and communication costs of the key-value cache during autoregressive decoding.[3]

Llama 2-Chat added supervised fine-tuning and reinforcement learning from human feedback. Meta collected preference comparisons, trained separate reward models for helpfulness and safety, and used rejection sampling and Proximal Policy Optimization during successive alignment rounds. The paper also described red-team exercises and safety evaluations. Those results applied to the tested Chat checkpoints and prompts, not to every later fine-tune built from the base weights.[3]

The release also changed access. The Llama 2 Community License granted a royalty-free limited right to use, modify, and redistribute the materials, subject to its conditions and an acceptable use policy. It also required organizations above its 700 million monthly active user threshold on the release date to request a separate license from Meta.[4]

### Llama 3 and the 3.x releases

The April 2024 Llama 3 release kept a dense autoregressive [Transformer](https://aiwiki.ai/wiki/transformers) but changed its tokenizer and attention configuration. Its 128,000-token vocabulary combined a 100,000-token tokenizer base with 28,000 tokens intended to improve non-English coverage. Both released sizes used grouped-query attention with eight key-value heads. Meta's later Llama 3 technical report says that most gains came from data quality, data diversity, and training scale rather than a new core architecture.[5]

Llama 3.1 completed that training line with 8B, 70B, and 405B checkpoints. Meta trained the 405B model on 15.6 trillion text tokens, first with an 8K context and then through continued pretraining that increased the context to 128K. The report describes a high-level final data mix of about 50 percent general knowledge, 25 percent mathematical and reasoning material, 17 percent code, and 8 percent multilingual data. It does not publish a complete list of training documents.[5]

The 3.1 post-training process used repeated rounds of supervised fine-tuning and [Direct Preference Optimization](https://aiwiki.ai/wiki/direct_preference_optimization_dpo). It added documented support for eight languages and tool-use formats. The associated license also permitted using model outputs to improve other models, a use that the Llama 2 license had restricted.[7][8]

Llama 3.2 split development in two directions. The 1B and 3B text models were produced by pruning larger Llama 3.1 models and using logits from 8B and 70B teachers as token-level targets. Meta's model card reports up to nine trillion pretraining tokens for these small models. The 11B and 90B vision releases instead attached a separately trained vision encoder to a Llama 3.1 language model through cross-attention layers. Their image-text training used six billion image-text pairs, and the model card lists English as the supported language for combined image-and-text use.[9][10]

Llama 3.3 was a 70B text checkpoint focused on the instruction-tuned line. Its model card lists a 128K context, eight supported languages, more than 15 trillion pretraining tokens, and more than 25 million synthetic examples in fine-tuning. A comparison to Llama 3.1 405B depends on task, prompt, and serving configuration; the smaller parameter count does not make the two checkpoints interchangeable.[11]

### Llama 4

Llama 4 changed both the model architecture and the training data description. Scout and Maverick are autoregressive, natively multimodal models that take text and images as input and produce text or code. They use early fusion, which mixes modality representations during model training, rather than adding a vision adapter to a finished text model.[12]

Both released models activate about 17 billion parameters for a token, but their total parameter counts differ. Scout has 16 experts and about 109 billion total parameters. Maverick has 128 experts and about 400 billion total parameters. A router directs computation through a subset of expert parameters, so "17B active" describes per-token computation while "109B total" or "400B total" describes the stored model capacity. This is not directly comparable to the parameter count of a dense model, where nearly all model parameters participate in each token computation.[12]

Meta documents a 10 million-token context window for Scout and one million tokens for Maverick. Its model card reports about 40 trillion pretraining tokens for Scout and 22 trillion for Maverick. The stated data sources include public data, licensed data, publicly shared Facebook and Instagram posts, and interactions with Meta AI. That disclosure differs from Llama 2's statement that its corpus used publicly available sources and no Meta user data.[3][12]

Meta's launch article also discussed Behemoth, a 288-billion-active-parameter teacher model that was still in training. The public April 2025 release and the official model repository list Scout and Maverick, not Behemoth, as downloadable Llama 4 checkpoints. Claims about Behemoth in the launch material were therefore claims about an unreleased development model, not results that users could reproduce from released weights.[6][18]

## Architecture and training

### Dense Transformer generations

The text-only checkpoints from LLaMA through Llama 3.3 are decoder-only, autoregressive language models. They predict the next token from earlier tokens and generate by repeating that operation. The first generation adapted a standard Transformer in three well documented ways:

- Pre-normalization with RMSNorm applies normalization before each Transformer sublayer.
- SwiGLU replaces the original Transformer's ReLU feed-forward activation.
- [Rotary Position Embedding](https://aiwiki.ai/wiki/rotary_position_embedding) applies position-dependent rotations inside attention rather than adding learned absolute position vectors.[1]

These choices persisted through the dense generations. Later changes addressed serving and scale. Llama 2 introduced grouped-query attention in the released 70B model. Llama 3 used it in all three documented sizes, including the 405B model, and expanded the tokenizer vocabulary. Long-context support in Llama 3.1 came from continued pretraining with progressively longer sequences, not merely from changing a configuration value.[3][5]

### Vision and mixture-of-experts

The multimodal releases use two different designs. Llama 3.2 Vision preserves a pretrained language-model core and feeds vision-encoder representations into it through cross-attention adapter layers. The language model and vision components therefore have distinct training histories.[10]

Llama 4 uses a [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) model and early fusion for native multimodality. Expert routing reduces the parameters used on each token, but the full checkpoint still has to store the total parameters. Memory requirements depend on weight precision, quantization, cache size, context length, batch size, and serving software. The active-parameter number alone is not a hardware requirement.[12]

### Pretraining and post-training are separate

Pretraining teaches token prediction from large corpora. Post-training changes how a model responds to prompts and which outputs it prefers. Across the family, Meta used different combinations of [instruction tuning](https://aiwiki.ai/wiki/instruction_tuning), rejection sampling, [reinforcement learning from human feedback](https://aiwiki.ai/wiki/rlhf), direct preference optimization, and synthetic data.[3][5][9][12]

The distinction matters when reading a model card or benchmark table. A base checkpoint can have strong language-model benchmark scores while remaining difficult to use as an assistant. An Instruct checkpoint may follow directions and refuse some requests because of its post-training. A safety result for an Instruct checkpoint does not transfer automatically to its base model, a derivative fine-tune, or a quantized build.

## Licensing and openness

Llama is commonly described as "open source", including in Meta announcements. "Open-weight" is the more precise description of the model family. Users can obtain checkpoints and inspect or modify them, but each generation is distributed under a custom agreement with conditions that go beyond standard permissive software licenses.[4][8][13]

The Open Source Initiative's Open Source AI Definition requires freedom to use, study, modify, and share an AI system, along with access to the preferred form for making modifications. That includes training and data-processing code, sufficiently detailed data information, and parameters under appropriate terms. OSI's FAQ specifically lists Llama 2 among systems that do not pass because required components are missing or their legal terms conflict with open-source principles.[14][15]

The practical license position also varies by generation:

- The original LLaMA release was gated and noncommercial.
- Llama 2 granted many research and commercial uses but restricted using its materials or outputs to improve a non-Llama large language model.
- Llama 3.1 and later licenses allow outputs to be used to improve other models, while requiring specified attribution and naming for distributed derivatives.
- The community licenses incorporate an acceptable use policy and retain the 700 million monthly active user threshold for certain large organizations.[2][4][8][13]

Developers need the license attached to the exact checkpoint they use. A repository label, model-hosting tag, or provider's marketing category is not a substitute for those terms. The license governs the Meta materials; a deployment may also involve third-party code licenses, data rights, privacy duties, and laws that depend on the application and jurisdiction.

Open weights also do not make the training process fully reproducible. Meta publishes architecture descriptions, training-token totals, selected data categories, evaluation results, model cards, and some inference tools. It does not publish the complete training corpus or all production training code. The December 2025 Foundation Model Transparency Index classified Meta as an open-weight developer but placed it in the lower half of the companies it assessed; the report identified training-data acquisition and properties as areas of continuing opacity across developers.[16]

## Evaluation and comparison

Meta's papers and model cards report results on academic tests, human preference studies, safety suites, tool-use tasks, and multimodal benchmarks. These tables document how a particular checkpoint performed in a stated setup. They are not timeless rankings of the Llama family.

Several details can change a result:

- Base, Chat, and Instruct checkpoints optimize for different behavior.
- Prompt templates, few-shot examples, sampling settings, tool definitions, and grading code affect scores.
- Full-precision, quantized, fine-tuned, and provider-hosted versions may not match.
- Public benchmark items can appear in web-scale training corpora. The Llama 2 and Llama 3 papers each include [data contamination](https://aiwiki.ai/wiki/data_contamination) analyses, but both also describe limits to detecting or estimating contamination.[3][5]
- A long-context model can accept a documented maximum sequence without retrieving or reasoning over every part of that sequence reliably.

The Holistic Evaluation of Language Models project was created partly because model reports often used different scenarios and metrics. It recommends standardized conditions, multiple metrics, and disclosure of prompts and outputs rather than reducing a model to one aggregate score.[17] A useful comparison should therefore name the exact checkpoint, weight precision, runtime, prompt format, dataset version, metric, and evaluation date.

Llama 4 adds another checkpoint-specific caution. Meta provides quantized weights for deployment, but its model card says the reported benchmark and safety evaluations used bfloat16 models. The card's comparison tables should not be presented as measured results for every released quantization.[12]

## Safety, limitations, and deployment

The family shares the normal failure modes of generative language models, including inaccurate statements, fabricated citations, biased associations, unsafe completions, and prompt-dependent behavior. The original paper measured toxicity, stereotypes, and truthfulness and stated that its models were likely to generate incorrect answers. These findings were properties to investigate, not guarantees that later models eliminated the same problems.[1]

Meta expanded model-level and system-level [AI safety](https://aiwiki.ai/wiki/ai_safety) work in later generations. The Llama 2 paper describes safety-specific preference data, iterative tuning, and [red teaming](https://aiwiki.ai/wiki/red_teaming). It also says its testing focused on English and could not cover every scenario. The Llama 3 report added multilingual and tool-use testing and released Llama Guard 3, Prompt Guard, and Code Shield as optional system components. Its own measurements showed a tradeoff: filtering reduced policy violations in the tested suites while increasing refusals on benign prompts.[3][5]

These safeguards solve different problems. A content classifier can screen prompts or outputs against a policy. A prompt-attack detector can identify some jailbreaks or [prompt injection](https://aiwiki.ai/wiki/prompt_injection). Static analysis can find some insecure generated code. None of them makes model output true, authorizes an external tool action, or proves that an application is safe. Meta's Llama 4 model card says the models should be deployed as part of a system with use-case-specific protections and that testing cannot cover all scenarios.[12]

A deployment review should separate at least four layers:

1. **Checkpoint:** Confirm the generation, size, base or Instruct status, precision, context limit, supported modalities, languages, and knowledge cutoff in the applicable [model card](https://aiwiki.ai/wiki/model_card).
2. **Runtime:** Test the actual tokenizer, prompt template, quantization, serving engine, and hardware limits. A model downloaded through [Hugging Face](https://aiwiki.ai/wiki/hugging_face) may be packaged differently from Meta's native checkpoint.
3. **Application:** Evaluate retrieval sources, tool permissions, identity and access controls, logging, privacy, fallback behavior, and human review for consequential actions. If the checkpoint is placed inside an [AI agent](https://aiwiki.ai/wiki/ai_agents), evaluate the complete scaffold and action environment rather than the model alone.
4. **Observed behavior:** Measure task quality, [hallucination](https://aiwiki.ai/wiki/hallucination), harmful output, false refusals, security failures, latency, and resource use on representative inputs. Repeat the evaluation after changing the checkpoint or serving stack.

This process avoids treating a family name as a capability or safety specification. "Llama" identifies a lineage of models. The exact checkpoint, license, runtime, and surrounding system determine what a deployed application can do and which risks remain.

## References

1. Hugo Touvron et al., ["LLaMA: Open and Efficient Foundation Language Models"](https://arxiv.org/abs/2302.13971). arXiv, 2023.
2. Meta AI, ["Introducing LLaMA: A foundational, 65-billion-parameter large language model"](https://ai.meta.com/blog/large-language-model-llama-meta-ai/). February 24, 2023.
3. Hugo Touvron et al., ["Llama 2: Open Foundation and Fine-Tuned Chat Models"](https://arxiv.org/abs/2307.09288). arXiv, 2023.
4. Meta, ["Llama 2 Community License Agreement"](https://github.com/meta-llama/llama/blob/main/LICENSE). July 18, 2023.
5. Aaron Grattafiori et al., ["The Llama 3 Herd of Models"](https://arxiv.org/abs/2407.21783). arXiv, 2024.
6. Meta, ["Llama models repository"](https://github.com/meta-llama/llama-models). GitHub.
7. Meta AI, ["Introducing Llama 3.1: Our most capable models to date"](https://ai.meta.com/blog/meta-llama-3-1/). July 23, 2024.
8. Meta, ["Llama 3.1 Community License Agreement"](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE). July 23, 2024.
9. Meta, ["Llama 3.2 text model card"](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md). GitHub.
10. Meta, ["Llama 3.2 Vision model card"](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md). GitHub.
11. Meta, ["Llama 3.3 model card"](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md). GitHub.
12. Meta, ["Llama 4 model card"](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md). GitHub.
13. Meta, ["Llama 4 Community License Agreement"](https://github.com/meta-llama/llama-models/blob/main/models/llama4/LICENSE). April 5, 2025.
14. Open Source Initiative, ["The Open Source AI Definition 1.0"](https://opensource.org/ai/open-source-ai-definition). October 2024.
15. Open Source Initiative, ["Open Source AI Definition FAQs"](https://opensource.org/ai/faq). Accessed July 28, 2026.
16. Alexander Wan et al., ["The 2025 Foundation Model Transparency Index"](https://crfm.stanford.edu/fmti/December-2025/paper.pdf). Stanford Center for Research on Foundation Models, 2025.
17. Percy Liang et al., ["Holistic Evaluation of Language Models"](https://arxiv.org/abs/2211.09110). Transactions on Machine Learning Research, 2023.
18. Meta AI, ["The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation"](https://ai.meta.com/blog/llama-4-multimodal-intelligence/). April 5, 2025.