Llama 3

Llama 3 is a family of open-weight large language models developed by Meta. Meta released the original Llama 3 checkpoints on April 18, 2024, in 8-billion-parameter and 70-billion-parameter sizes. Each size was distributed as a pretrained base model and as an instruction-tuned model intended for assistant-style dialogue.^[1][2] The April models generate text and code, accept text-only input, and support sequences of up to 8,192 tokens.^[2]

Llama 3 retained the dense, decoder-only Transformer design of Llama 2, but used a much larger tokenizer, grouped-query attention at both model sizes, and more than 15 trillion pretraining tokens.^[1][2] Meta described the models as openly available and called them "open source" at launch, but the weights are governed by a custom community license rather than an open-source license approved by the Open Source Initiative.^[3][4]

This article concerns the four checkpoints released in April 2024. It does not treat the 405B model, 128,000-token context, explicit multilingual support, or built-in tool use introduced with Llama 3.1 in July 2024 as capabilities of the original release. Meta's later technical paper makes the same distinction and states that its reported evaluation results are for Llama 3.1 models.^[5]

Release

Meta announced Llama 3 on April 18, 2024. The release contained the following checkpoints:^[1][2]

Scroll sideways for more →

The base checkpoints were released for adaptation to natural-language generation tasks and were not fine-tuned for question answering or conversation. The Instruct checkpoints used a structured chat format with special tokens for the beginning of text, message headers, roles, and the end of each turn. Meta's original repository required users to format dialogue according to that template to obtain the intended behavior.^[2][6]

Meta released the weights through its download service and gated repositories on Hugging Face. The launch post also announced planned availability through cloud and model-serving partners including AWS, Databricks, Google Cloud, IBM watsonx, Microsoft Azure, NVIDIA NIM, and Snowflake. These distribution arrangements made the same checkpoint family available for local deployment and managed hosting, subject to the license and acceptable-use policy.^[1][6]

The model card assigns different data cutoffs to the two sizes: March 2023 for 8B and December 2023 for 70B. The models are static checkpoints trained on offline data, so those dates are data-freshness boundaries rather than guarantees about every fact represented in their weights.^[2]

Architecture

Llama 3 is an autoregressive, dense, decoder-only Transformer. The later Llama 3 technical report says that the architecture differs only modestly from earlier Llama models and attributes most of the quality improvement to data quality, data diversity, and training scale rather than a new network type.^[5] The report gives the following dimensions for the 8B and 70B backbones:

Scroll sideways for more →

Grouped-query attention

Both original sizes use grouped-query attention with eight key-value heads. In the 8B model, 32 query heads share those eight key-value heads; in the 70B model, 64 query heads share the same number of key-value heads.^[5] GQA is an intermediate design between standard multi-head attention, in which every query head has its own key and value projections, and multi-query attention, in which all query heads share one key-value head. The technique reduces the size of the key-value cache used during autoregressive decoding while seeking to preserve model quality.^[7]

Llama 2 had used GQA only in its 34B and 70B configurations, whereas Meta applied it to both released Llama 3 sizes. Meta reported that the new tokenizer and GQA allowed Llama 3 8B to maintain inference efficiency comparable to Llama 2 7B despite having about one billion more parameters.^[1][8]

Tokenizer and position encoding

Meta described the Llama 3 tokenizer as having a 128,000-token lexical vocabulary. The later technical report says it combines about 100,000 tokens from the tiktoken tokenizer with 28,000 additional tokens selected to improve support for non-English languages. On Meta's English sample, the new tokenizer increased the average characters represented per token from 3.17 for Llama 2 to 3.94 for Llama 3. The launch post separately reported up to 15 percent fewer tokens than Llama 2 on its tokenizer benchmarks.^[1][5]

The models use rotary position embedding, or RoPE, with a base-frequency parameter of 500,000.^[5] RoPE represents token positions through rotations applied within attention and thereby introduces relative-position information into attention scores.^[9] The April checkpoints were nevertheless trained and released with an 8,192-token sequence limit. The 128,000-token context later associated with the Llama 3 family belongs to Llama 3.1, not to these original checkpoints.^[2][5]

Meta also used an attention mask that prevented tokens from attending across document boundaries when multiple documents were packed into one training sequence. The launch account says the models were trained on sequences of 8,192 tokens with this boundary mask.^[1][5]

Training

Pretraining data

Meta reported that Llama 3 was pretrained on more than 15 trillion tokens collected from publicly available sources. It described the corpus as about seven times the size of the Llama 2 corpus and containing about four times as much code. More than 5 percent of the corpus consisted of non-English data covering over 30 languages, although Meta cautioned that the April checkpoints were intended for English and were not expected to match their English performance in other languages.^[1][10]

The launch documentation describes a filtering pipeline that combined heuristic filters, filters for sexually explicit material, semantic deduplication, and learned text-quality classifiers. Meta said that it used Llama 2 to help generate training labels for the quality classifiers used to curate Llama 3 data. It also reported excluding or removing sources known to contain a high volume of personal information about private individuals and said that neither the pretraining nor fine-tuning data included Meta user data. These are disclosures by the developer; Meta did not publish the underlying training corpus or a source-by-source data inventory for independent reproduction.^[2][10]

Meta's later technical report describes the final family-wide pretraining mix as approximately 50 percent general-knowledge tokens, 25 percent mathematics and reasoning tokens, 17 percent code tokens, and 8 percent multilingual tokens. It says the team selected the mix through smaller scaling-law experiments and used specialized classifiers and quality ranking to rebalance web data.^[5] Because that report accompanied Llama 3.1 and states that its result tables use Llama 3.1, its detailed training account is useful documentation of the shared Llama 3 development program but should not be read as an April benchmark table.^[5]

Training scale and compute

Meta's model card reports 1.3 million H100-80GB GPU-hours for Llama 3 8B and 6.4 million H100-80GB GPU-hours for Llama 3 70B, for a cumulative 7.7 million GPU-hours. It estimates 390 metric tons of carbon-dioxide equivalent for the 8B run and 1,900 for the 70B run, totaling 2,290 tCO2e. Meta states that its sustainability program offset 100 percent of those estimated emissions.^[2]

The launch post says that both the 8B and 70B models continued improving after training on as many as 15 trillion tokens, far beyond the roughly 200 billion tokens Meta associated with a compute-optimal 8B model under its comparison to Chinchilla scaling. Meta's design goal was to spend more compute during training so that smaller released models would offer better quality at a given inference cost.^[1] This is a developer interpretation of its scaling experiments rather than a general rule that every model benefits equally from the same degree of additional training.

Instruction tuning

The Instruct variants were produced from the pretrained checkpoints through post-training. Meta's April launch account explicitly lists four components: supervised fine-tuning, rejection sampling, proximal policy optimization, and direct preference optimization. It says that prompt quality and human preference rankings had a large effect on the final behavior and that multiple rounds of annotation quality assurance were used.^[1]

The model card summarizes the tuned models more broadly as using supervised fine-tuning and reinforcement learning from human feedback. It reports that the fine-tuning data included public instruction datasets and more than 10 million human-annotated examples.^[2] A later technical report describes a revised family-wide pipeline centered on supervised fine-tuning, rejection sampling, and DPO, but that later description should not be substituted for the launch-era statement that PPO was also part of the April recipe.^[1][5]

Evaluation

Developer-reported benchmarks

Meta evaluated the released checkpoints with an internal evaluation library and published the prompting details separately. Results therefore depend on the stated number of examples, prompt format, answer extraction, and scoring procedure and should not be compared with results produced under different protocols as if they were directly interchangeable.^[2][11]

Meta reported the following automatic benchmark results for the pretrained checkpoints:^[2]

Scroll sideways for more →

For the instruction-tuned checkpoints, Meta reported:^[2]

Scroll sideways for more →

Meta also assembled a private human-evaluation set of 1,800 prompts, with 150 prompts in each of 12 categories including coding, mathematical reasoning, advice, brainstorming, question answering, creative writing, extraction, role play, rewriting, and summarization. Meta said that its modeling teams could not access the set during development and reported that human preference rankings favored the 70B Instruct model over several contemporaneous comparison systems.^[1][11] Because the prompt set and full response-level data were not released in the launch materials, the aggregate preference claim was not independently reproducible from those materials.

The later paper's tables must not be used as April Llama 3 scores. Its overview distinguishes the April 8B and 70B checkpoints from the July Llama 3.1 checkpoints and states, "All results in this paper are for the Llama 3.1 models."^[5]

Independent and safety-focused studies

Independent results illustrate why broad capability claims need task and protocol boundaries. An ArabicNLP 2024 study evaluated the original Llama 3 70B on a collection of Modern Standard Arabic and dialectal Arabic generation tasks. The authors reported that it lagged the ChatGPT system they tested and smaller Arabic-specialized models on their selected tasks. They also noted limitations including model-version drift in the comparison API, prompt sensitivity, and the absence of other open models from the comparison.^[12] These results are consistent with Meta's own launch warning that the April release was intended for English despite containing some multilingual pretraining data.^[1][2]

CyberSecEval 2, a Meta-authored evaluation suite released with Llama 3, included Llama 3 70B Instruct in tests of cyberattack assistance, false refusals, prompt injection, code-interpreter abuse, and vulnerability exploitation. Across the models tested, the authors observed at least a 26 percent prompt-injection success rate and concluded that prompt injection remained unsolved. The paper also found that stronger safety conditioning could trade off against compliance with benign but security-related requests.^[13] These findings concern an evaluated system and test suite, not a guarantee about every deployment configuration.

AIR-Bench 2024 independently included both original Llama 3 Instruct sizes among 22 models tested on 5,694 prompts derived from regulatory and company-policy risk categories. Its primary metric was refusal rate, judged with category-specific automatic graders that the authors checked against a smaller human sample. The study found substantial variation by model and risk category and cautioned that high refusal can also reflect over-cautious behavior rather than useful safety in every application.^[14]

A July 2024 preprint tested whether access to the Llama 3 Instruct weights made safety tuning removable. Its author reported that parameter-efficient fine-tuning and representation-editing methods could sharply reduce refusals in minutes on a single GPU while preserving scores on a small set of general benchmarks.^[15] The paper also explicitly warns that its attack-success metric measures whether the model attempts a prohibited response, not whether that response is accurate or useful, and that the work was a first-version preprint. The bounded conclusion is that weight access permits downstream actors to alter refusal behavior; the headline timings are not a comprehensive measurement of all safety properties.^[15]

Safety and limitations

Meta described safety as a property of the full application rather than of the model weights alone. The Instruct models underwent internal and external red teaming, including testing related to chemical and biological hazards, cyberattacks, child safety, and false refusals. Meta released Llama Guard 2 for classifying prompts and responses, Code Shield for filtering insecure generated code, and CyberSecEval 2 for evaluation. It recommended application-specific input and output filtering, testing, monitoring, and additional safeguards.^[1][10][13]

The model card says that testing had primarily been conducted in English and could not cover all scenarios. It warns that the models can produce inaccurate, biased, or otherwise objectionable output and directs developers to perform safety testing and tuning for the intended application.^[2] The base checkpoints have no dialogue-specific safety alignment, while the Instruct checkpoints' refusal behavior can be changed through further fine-tuning. Neither variant should be treated as a factual database, a current information service, or a complete safety system.^[2][15]

The separate March and December 2023 data cutoffs limit knowledge of later events.^[2] The 8,192-token limit also applies to the complete formatted sequence, including system messages, user input, dialogue history, and generated tokens. Applications that need current facts, long documents, or controlled access to external tools must add retrieval, truncation, tool execution, and validation at the system level rather than assume those features are present in the checkpoint.^[2][6]

License and openness

The April weights and accompanying software were distributed under the Meta Llama 3 Community License Agreement. It grants a worldwide, royalty-free, non-transferable limited license to use, reproduce, distribute, modify, and create derivative works from the Llama materials, subject to its conditions.^[3]

Among those conditions, distributors must provide the agreement, display "Built with Meta Llama 3," preserve a specified notice, and begin the name of a distributed model created from the materials with "Llama 3." The license incorporates Meta's acceptable-use policy and prohibits using the Llama materials or their outputs to improve another large language model, except a Llama 3 derivative. An organization whose products or services had more than 700 million monthly active users in the month before the release date must obtain a separate license from Meta before exercising the granted rights.^[3][16]

The acceptable-use policy prohibits categories of use including unlawful activity, child exploitation, certain weapons and military applications, malicious code, unauthorized professional practice, some processing of sensitive personal data, fraud, spam, impersonation, and failure to disclose known dangers of an AI system.^[16] The license also permits Meta to terminate rights for breach and requires the user to delete and stop using the materials after termination.^[3]

These restrictions are why "open-weight" is more precise than "open source" for the April Llama 3 release. The Open Source Initiative has argued that the Llama license fails the Open Source Definition because it restricts who may use the materials and the fields in which they may be used.^[4] The weights are inspectable and modifiable by authorized users, but weight availability does not by itself make the model or license open source.

Applications and later development

The release coincided with a major expansion of Meta AI. Meta said that the assistant was built with Llama 3 and made it available through Facebook, Instagram, WhatsApp, Messenger, and a standalone meta.ai website. At launch, Meta AI was being rolled out in English to more than a dozen countries and combined model responses with web search and separate image-generation features.^[17] The search and image systems were product-level components, not capabilities encoded solely in the released text checkpoints.

Meta released Llama 3.1 on July 23, 2024. That update added a 405B model, extended the supported context to 128,000 tokens, and added explicit multilingual and tool-use capabilities to designated checkpoints.^[5] Later Llama 3.x releases added other sizes and modalities. Those releases share a lineage and some architectural components with the April models, but their specifications, licenses, evaluation results, and safety profiles must be checked separately.

The April release paired downloadable 8B and 70B weights with a corpus Meta reported as more than 15 trillion tokens, GQA at both sizes, a 128,000-token lexical vocabulary, a documented chat template, and same-day use of Llama 3 in Meta AI.^{[1][2][5][6][17]} The underlying corpus was not published, the launch benchmark tables were developer-run, the license imposes use restrictions, and application safety depends on system-level controls outside the base model.^{[1][2][3][10]}

See also

• LLaMA
• Code Llama
• Llama Guard
• Open-source AI
• Byte-pair encoding
• Context window

References

1. ^Meta AI, "Introducing Meta Llama 3: The most capable openly available LLM to date," April 18, 2024. ai.meta.com/...meta-llama-3
2. ^Meta, "Llama 3 Model Card," April 18, 2024. github.com/...MODEL_CARD.md
3. ^Meta, "Meta Llama 3 Community License Agreement," April 18, 2024. github.com/...LICENSE
4. ^Open Source Initiative, "Meta's LLaMa license is still not Open Source," February 18, 2025. opensource.org/...license-is-still-not-open-source
5. ^Llama Team, AI at Meta, "The Llama 3 Herd of Models," arXiv:2407.21783, July 2024, revised November 2024. arxiv.org/...2407.21783
6. ^Meta, "Meta Llama 3 repository and inference guide," April 2024. github.com/...llama3
7. ^Joshua Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," arXiv:2305.13245, May 2023. arxiv.org/...2305.13245
8. ^Hugo Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv:2307.09288, July 2023. arxiv.org/...2307.09288
9. ^Jianlin Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding," arXiv:2104.09864, April 2021. arxiv.org/...2104.09864
10. ^Meta AI, "Our responsible approach to Meta AI and Meta Llama 3," April 18, 2024. ai.meta.com/...meta-llama-3-meta-ai-responsibility
11. ^Meta, "Llama 3 Evaluation Details," April 2024. github.com/...eval_details.md
12. ^Md Tawkat Islam Khondaker et al., "Benchmarking LLaMA-3 on Arabic Language Generation Tasks," Proceedings of the Second Arabic Natural Language Processing Conference, August 2024. aclanthology.org/2024.arabicnlp-1.24
13. ^Manish Bhatt et al., "CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models," arXiv:2404.13161, April 2024. arxiv.org/...2404.13161
14. ^Yi Zeng et al., "AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies," arXiv:2407.17436, July 2024, revised August 2024. arxiv.org/...2407.17436
15. ^Dmitrii Volkov, "BadLlama 3: removing safety finetuning from Llama 3 in minutes," arXiv:2407.01376v1, July 2024. arxiv.org/...2407.01376
16. ^Meta, "Meta Llama 3 Acceptable Use Policy," April 2024. github.com/...USE_POLICY.md
17. ^Meta Newsroom, "Meet Your New Assistant: Meta AI, Built With Llama 3," April 18, 2024. about.fb.com/...meta-ai-assistant-built-with-llama-3