Llama 2

Llama 2 is a family of open-weight large language models developed by Meta AI. Meta released pretrained and dialogue-tuned checkpoints with 7 billion, 13 billion, and 70 billion parameters on July 18, 2023. The dialogue models are called Llama 2-Chat. Meta also trained 34B base and chat models and reported some of their results, but withheld those checkpoints because it had not completed enough red teaming before release.^[1]

All four sizes use a decoder-only Transformer, a 4,096-token context window, and a 32,000-token SentencePiece vocabulary. The 34B and 70B configurations use grouped-query attention, while the 7B and 13B configurations use ordinary multi-head attention.^[1] Every base model was pretrained on 2 trillion tokens from what Meta described as publicly available online sources. The paper reports that about 89.7 percent of the classified pretraining data was English, and the official model card identifies September 2022 as the pretraining-data cutoff.^[1][2]

Meta described the release as open source and made the weights available for research and many commercial uses. The weights are more precisely described as open-weight because the custom Llama 2 Community License imposes use and user restrictions that are not part of an Open Source Initiative approved license.^[3][4] Llama 2 is a static historical model family. Meta introduced Llama 3 as its successor on April 18, 2024.^[5]

Release and model family

Meta and Microsoft announced Llama 2 jointly on July 18, 2023. Meta called Microsoft its preferred partner and distributed the models through its own download process and Hugging Face. Microsoft placed the 7B, 13B, and 70B models in the Azure AI model catalog and announced support for running them locally on Windows through ONNX Runtime and DirectML. Meta also named Amazon Web Services and other providers as distribution channels.^[6][7]

The release separated general pretrained models from chat-oriented models:

Scroll sideways for more →

The base checkpoints are not instruction-following assistants. Meta's inference documentation describes their output as a natural continuation of the input text. Llama 2-Chat expects a specific dialogue template containing instruction, system-message, beginning-of-sequence, and end-of-sequence markers. Omitting or changing that format can change behavior and makes comparisons with the documented model less reliable.^[7]

The model card says that Llama 2 was trained between January and July 2023. It gives a September 2022 cutoff for pretraining data and says some tuning data extended to July 2023.^[2] These dates limit the information available to the static checkpoints. They do not guarantee that the model learned every fact published before the cutoff, and the model has no built-in mechanism for updating knowledge after training.

Meta's original meta-llama/llama repository is now marked deprecated. It remains a record of the Llama 2 prompt format and original inference examples, but Meta directs current users to the consolidated llama-models repository. That repository continues to list Llama 2 as a 7B, 13B, and 70B family with a 4K context and provides a command for showing older model versions.^[7][8]

Architecture

Llama 2 follows the architecture established by its predecessor, LLaMA. It is an autoregressive, decoder-only Transformer that predicts the next token from preceding tokens. It applies RMSNorm before attention and feed-forward sublayers, uses SwiGLU activations, and represents position with rotary position embedding, or RoPE.^[1][9]

The most visible changes from Llama 1 were a doubling of the context window from 2,048 to 4,096 tokens, a 40 percent increase in the number of pretraining tokens, revised data cleaning and mixing, and GQA in the two larger configurations. The 34B and 70B models share key-value projections across groups of query heads. This reduces the size of the key-value cache during autoregressive generation relative to standard multi-head attention and was intended to improve inference scalability.^[1]

Scroll sideways for more →

The tokenizer is the same basic tokenizer used by Llama 1: byte-pair encoding implemented with SentencePiece, with a 32,000-token vocabulary. Numbers are split into individual digits, and unknown UTF-8 characters fall back to byte representations.^[1] The model accepts text and generates text. It has no native image, audio, video, retrieval, browsing, or tool-execution interface.

The 4,096-token limit applies to the whole formatted sequence, not separately to the prompt and answer. For Llama 2-Chat this includes the system message, prior turns, current user message, template tokens, and generated response. Applications that retain long conversations therefore need a policy for truncating or summarizing history.

Pretraining

Data

Meta says it trained Llama 2 on 2 trillion tokens from a new mixture of publicly available online data. The company did not publish the corpus or a source-by-source inventory. It reported more robust data cleaning, upsampling of sources considered more factual, and an effort to remove sites known to contain a high volume of personal information about private individuals. Meta also states that neither the pretraining nor fine-tuning data included Meta user data.^[1][2]

The paper says no additional safety filtering was applied to the pretraining mixture. Its stated rationale was that aggressive removal could erase material needed for tasks such as hate-speech classification and could disproportionately remove references to demographic groups. This choice also means that the base models can reproduce harmful, biased, offensive, or false material and require task-specific evaluation and mitigation before deployment.^[1]

Meta used language identification to estimate the composition of the corpus. The paper reports 89.70 percent English and 8.38 percent "unknown," with the unknown group partly consisting of programming code. Every individually listed non-English language accounted for less than 0.2 percent. Meta consequently framed English as the intended language and warned that performance in other languages was fragile.^[1][2]

Optimization and hardware

All sizes used AdamW with beta values of 0.9 and 0.95, a cosine learning-rate schedule, 2,000 warmup steps, weight decay of 0.1, gradient clipping at 1.0, and a global batch size of 4 million tokens. The paper says training loss continued to decline through 2 trillion processed tokens rather than showing saturation within the run.^[1]

Pretraining ran on Meta's Research SuperCluster and internal production clusters using NVIDIA A100 80 GB GPUs. The two environments used different network interconnects and 350 W or 400 W power caps. The paper's carbon accounting covers estimated pretraining electricity-related emissions. It excludes hardware manufacture and some non-GPU power demands, and it uses peak device power rather than measured energy at every moment.^[1]

Scroll sideways for more →

The 34B row is necessary to reconcile the total even though that checkpoint was not released. Meta reported offsetting all 539 tCO2e through its sustainability program. An offset does not make the physical training emissions zero, and the estimate is not a lifecycle assessment of the hardware, data centers, later fine-tuning, or inference.^[1]

Llama 2-Chat post-training

Supervised fine-tuning

Meta began with public instruction-tuning data and then commissioned its own dialogue demonstrations. The researchers reported that a relatively small set of carefully reviewed examples performed better than millions of lower-quality third-party examples in their experiments. They stopped collecting supervised examples after 27,540 annotations and fine-tuned for two epochs while calculating loss only on answer tokens.^[1]

The supervised stage produced an initial assistant model. It was followed by iterative reinforcement learning from human feedback, or RLHF. Annotators wrote prompts, compared two candidate responses, selected the preferred response, and graded the strength of that preference. The collection separated helpfulness judgments from safety judgments because a detailed answer can be useful while violating a safety policy.^[1]

Reward models and RLHF

Meta trained separate helpfulness and safety reward models. The paper reports 1,418,091 internally collected pairwise comparisons across safety and helpfulness. Together with several public preference datasets, the reported reward-model mixture contained 2,919,326 comparisons. The total is not a claim that all 2.9 million comparisons were newly collected by Meta.^[1]

The reward models assigned scalar scores to prompt-response pairs. Meta used those scores in two optimization paths:

• Rejection sampling generated multiple candidate answers and retained the answer with the highest reward-model score for further fine-tuning.
• Proximal Policy Optimization updated model parameters against the learned reward while constraining movement away from the preceding model.

Only the 70B model generated responses for rejection sampling in the paper's final procedure. Its selected outputs were used to fine-tune checkpoints of all sizes. PPO was applied across the 7B, 13B, 34B, and 70B configurations, although the 34B checkpoints were withheld.^[1]

Meta performed five successive RLHF rounds. Each new model changed the distribution of outputs, so new preference data was collected from later checkpoints to keep the reward models closer to the behavior they were scoring. The procedure does not establish a universal human preference: it optimizes against the annotation rules, prompts, raters, and reward models used in this development process.

Ghost Attention

Llama 2-Chat also used Ghost Attention, or GAtt, to improve adherence to a system instruction over multiple turns. During fine-tuning, Meta synthetically placed the controlling instruction beside later user messages, trained on that representation, and then removed the repeated instruction from the visible dialogue. The paper reports stronger retention of constraints such as a role, language, or formatting rule for roughly 20 or more turns in its tests, followed by degradation.^[1]

Ghost Attention is a post-training method rather than a longer context window. It does not change the 4,096-token architectural limit, and it does not ensure that every system instruction will be followed.

Evaluation

Base-model benchmarks

Meta evaluated the base models with an internal evaluation library. The paper grouped multiple tasks into category averages, used different shot counts across tasks, and in some comparisons selected the better of its reproduced result and a published result for another model. The figures below are therefore developer-reported results under Meta's stated protocols, not hardware-independent measures or directly interchangeable leaderboard values.^[1]

Scroll sideways for more →

The non-monotonic 34B math result is the value reported in the paper and should not be silently replaced by an assumed interpolation. Meta also compared 70B with reported closed-model scores, but differences in model versions, prompts, answer extraction, and evaluation code limit those comparisons.^[1]

Chat evaluation

For helpfulness, Meta asked three human annotators per prompt to compare model responses across roughly 4,000 single-turn and multi-turn prompts. Against gpt-3.5-turbo-0301, Llama 2-Chat 70B received a 36 percent win rate and a 31.5 percent tie rate in that study. The authors explicitly noted that the prompt set omitted coding and reasoning prompts, rated only the final response of a multi-turn conversation, and could produce different results under different prompts or instructions.^[1]

Meta separately used about 2,000 adversarial prompts for safety evaluation. It reported low violation rates for Llama 2-Chat relative to the comparison systems, but cautioned that its prompt set, content standards, rater judgments, and safety tuning could favor its own models. Multi-turn conversations produced more violations than single-turn conversations across the systems tested.^[1]

MT-Bench and Chatbot Arena were introduced as ways to evaluate open-ended chat responses and human preferences. Their authors found that automated LLM judges can exhibit position, verbosity, and self-enhancement biases. That evidence is a reason to preserve the protocol and judge identity around any chat score rather than treating a single number as an intrinsic property of Llama 2-Chat.^[10]

Truthfulness and toxicity benchmarks

Meta evaluated the base and chat models on TruthfulQA and ToxiGen. For the 70B base model it reported 50.18 percent truthful-and-informative generations on TruthfulQA and 24.60 percent classified-toxic generations on ToxiGen. For Llama 2-Chat 70B, the reported values were 64.14 percent and 0.01 percent. The chat models at 7B and 13B were reported at 0.00 percent classified-toxic generations on that ToxiGen setup.^[1]

These results show the behavior of specific automatic metrics under the paper's decoding settings. They do not demonstrate an absence of harmful output. A toxicity classifier can miss harmful associations, and a refusal can itself produce unequal quality of service.

Safety, bias, and limitations

Meta's safety pipeline combined supervised safety examples, a safety-specific reward model, adversarial preference data, rejection sampling, PPO, context distillation, and red teaming. The company tested categories including illicit activity, harmful activity, self-harm, and unqualified medical, financial, or legal advice. Meta reports internal and external red-team participation. Its red-team model outputs were in English, while prompts and dialogue contexts also included non-English material.^[1]

The paper nevertheless identifies several limits. Llama 2 can hallucinate, provide unqualified advice, produce inaccurate or biased content, and reflect its largely English pretraining data. The base checkpoints received no chat-specific safety alignment and need especially careful tuning. Llama 2-Chat can be over-cautious, declining benign requests or adding unnecessary safety text.^[1]

An ACL 2024 study examined all three Llama 2-Chat sizes and Llama 1 with 1,792 non-toxic prompts based on demographic associations in ToxiGen. The researchers manually categorized more than 20,000 outputs. They found that Llama 2-Chat largely avoided explicit harmful answers in this setup but sometimes produced harmful or unequal refusals, with refusal behavior varying across demographic name groups. The study was a focused case study with North America-centered demographic categories and human annotation, not a measurement of all social bias or all deployments.^[11]

Another study showed that access to weights can make refusal behavior changeable. In one ICLR 2024 workshop experiment, researchers used quantized low-rank adaptation on one GPU with a stated budget below $200. Their modified 70B Llama 2-Chat model refused about 1 percent of prompts on two refusal benchmarks while retaining similar scores on MMLU and HellaSwag. This result concerns a particular fine-tuning dataset, benchmark set, refusal detector, and attack procedure. It supports evaluating risks introduced by downstream fine-tuning, not the broader claim that every safety behavior can always be removed at that cost.^[12]

Meta's Responsible Use Guide treats the model as one component of a system. It recommends use-case risk assessment, held-out evaluation, human review where appropriate, red teaming, input and output filters, monitoring, and tests for prompt injection and privacy attacks. It also warns that safeguards can be subverted and that filters can trade safety against user experience.^[13]

Applications should not assume that Llama 2 is a factual database, current-information service, professional adviser, or complete safety system. The license and acceptable-use rules also do not technically prevent a local checkpoint from producing prohibited content. They are contractual and policy controls that operate alongside, not inside, the model weights.

License and openness

The Llama 2 Community License grants a non-exclusive, worldwide, non-transferable, royalty-free limited right to use, reproduce, distribute, modify, and create derivative works from the Llama materials.^[3] The grant is broad enough for many research and commercial uses, but it is conditional.

Key conditions include:

• distributors must provide the license and preserve Meta's specified attribution notice;
• use must comply with applicable law and the incorporated Llama 2 Acceptable Use Policy;
• the materials or their outputs may not be used to improve another large language model, except Llama 2 or its derivatives;
• an organization whose products or services had more than 700 million monthly active users in the month before the release must request a separate license from Meta;
• filing certain intellectual-property claims about the materials or their outputs terminates the granted licenses; and
• after termination for breach, the licensee must delete and cease using the materials.^[3]

The Acceptable Use Policy prohibits specified activities involving violence, terrorism, exploitation of children, human trafficking, illegal weapons, controlled substances, malware, unauthorized professional practice, certain personal-data processing, fraud, impersonation, and other harmful or unlawful uses.^[14] The policy categories are contractual restrictions. They are not evidence that every prohibited output is technically blocked.

Meta called Llama 2 open source at launch. The Open Source Initiative responded that the license does not satisfy the Open Source Definition because it restricts some users and fields of endeavor. "Open-weight" therefore communicates the central technical fact, that authorized users can download and modify weights, without implying that the full release satisfies an open-source licensing standard.^[4]

The 2023 Foundation Model Transparency Index assessed Meta through Llama 2 across 100 indicators covering upstream resources, the model, and downstream use. It placed Meta among the most transparent developers in the study while concluding that every assessed developer still had substantial room for improvement. In particular, the index found broad ecosystem-wide gaps around training-data creation, data labor, compute, downstream impact, and mechanisms for redress. Weight availability improved some forms of inspection but did not make the training corpus or full development process reproducible.^[15]

Derived models and successor

Llama 2 became the starting checkpoint for several Meta research releases, but those models have different training, contexts, and intended uses:

• Code Llama continued training Llama 2 on code-heavy data and added code completion, infilling in selected sizes, Python-specialized variants, instruction-tuned variants, and long-context training. The paper describes 7B, 13B, 34B, and 70B configurations. These capabilities should not be attributed to the unmodified Llama 2 checkpoints.^[16]
• Llama 2 Long was a research series produced through continual pretraining with longer sequences and 400 billion additional tokens. Its 7B and 13B variants were trained with 32,768-token sequences, while 34B and 70B used 16,384-token sequences. It is separate from the released 4,096-token Llama 2 family.^[17]
• Llama Guard was an input-output safety classifier instruction-tuned from Llama 2 7B. Its initial taxonomy covered six unsafe categories plus safe content and could be included in the prompt for adaptation. It was a separate safeguard model, not a safety layer embedded in Llama 2 itself.^[18]

Meta released Llama 3 on April 18, 2024, initially in 8B and 70B pretrained and instruction-tuned variants.^[5] Later Llama releases use different tokenizers, context windows, licenses, training data, and post-training procedures. Their specifications and evaluation results should not be applied retroactively to Llama 2.

Llama 2 remains relevant as a documented 2023 open-weight model family and as the base of later specialized research. Its documented strengths include a complete technical paper, model card, downloadable weights for eligible users, explicit training-compute estimates, and detailed alignment methodology. Its boundaries are equally important: a mostly English undisclosed corpus, a 4,096-token context, static 2022-era pretraining knowledge, custom licensing restrictions, developer-run benchmark tables, and safety behavior that depends on prompts, fine-tuning, and controls outside the checkpoint.

See also

• Foundation models
• Generative AI
• Model card
• Red teaming
• Quantization
• Hugging Face

References

1. ^Hugo Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv:2307.09288, July 2023. arxiv.org/...2307.09288
2. ^Meta, "Llama 2 Model Card," consolidated Llama Models repository. github.com/...MODEL_CARD.md
3. ^Meta, "Llama 2 Community License Agreement," July 18, 2023. ai.meta.com/...license
4. ^Stefano Maffulli, Open Source Initiative, "Meta's LLaMa license is not Open Source," July 20, 2023. opensource.org/...ama-2-license-is-not-open-source
5. ^Meta AI, "Introducing Meta Llama 3," April 18, 2024. ai.meta.com/...meta-llama-3
6. ^Meta, "Meta and Microsoft Introduce the Next Generation of Llama," July 18, 2023. about.fb.com/...llama-2
7. ^Meta, "Deprecated Llama 2 repository and inference guide." github.com/...llama
8. ^Meta, "Llama Models repository." github.com/...llama-models
9. ^Hugo Touvron et al., "LLaMA: Open and Efficient Foundation Language Models," arXiv:2302.13971, February 2023. arxiv.org/...2302.13971
10. ^Lianmin Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," arXiv:2306.05685, June 2023, revised December 2023. arxiv.org/...2306.05685
11. ^Khaoula Chehbouni et al., "From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards," Findings of ACL 2024. aclanthology.org/2024.findings-acl.927
12. ^Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish, "LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B," ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. arxiv.org/...2310.20624
13. ^Meta, "Responsible Use Guide: Building with Llama 2," July 2023. ai.meta.com/...responsible-use-guide
14. ^Meta, "Llama 2 Acceptable Use Policy." ai.meta.com/...use-policy
15. ^Rishi Bommasani et al., "The Foundation Model Transparency Index," arXiv:2310.12941, October 2023. arxiv.org/...2310.12941
16. ^Baptiste Roziere et al., "Code Llama: Open Foundation Models for Code," arXiv:2308.12950, August 2023, revised January 2024. arxiv.org/...2308.12950
17. ^Wenhan Xiong et al., "Effective Long-Context Scaling of Foundation Models," arXiv:2309.16039, September 2023, revised November 2023. arxiv.org/...2309.16039
18. ^Hakan Inan et al., "Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations," arXiv:2312.06674, December 2023. arxiv.org/...2312.06674