# Code Llama

> Source: https://aiwiki.ai/wiki/code_llama
> Updated: 2026-06-21
> Categories: AI Code Generation, Large Language Models, Meta AI, Open Source AI
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

**Code Llama** is a family of open-weight [large language models](/wiki/large_language_model) specialized for code generation and understanding, released by [Meta AI](/wiki/meta_ai) on August 24, 2023. Built on top of [Llama 2](/wiki/llama) and further trained on code-heavy datasets, Code Llama was, in Meta's words, "state-of-the-art for publicly available LLMs on code tasks" at launch, with its largest model (Code Llama-Instruct 70B, released January 2024) scoring 67.8% pass@1 on the HumanEval benchmark, slightly above [GPT-4](/wiki/gpt4)'s 67.0%.[1][2][5] The model family includes three variants (Code Llama base, Code Llama-Python, and Code Llama-Instruct) offered in four parameter sizes: 7B, 13B, 34B, and 70B.[1] Code Llama supports advanced capabilities such as fill-in-the-middle (FIM) code infilling, extended context windows of up to 100,000 tokens, and zero-shot instruction following for programming tasks.[1][2]

The accompanying research paper, "Code Llama: Open Foundation Models for Code," was authored by Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jeremy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.[1] The paper was first posted on [arXiv](/wiki/arxiv) on August 24, 2023, and was later published as a conference paper at the Conference on Language Modeling (COLM) in 2024.[1]

## What is Code Llama used for?

Code Llama is designed to assist with software development tasks, including generating code from natural language descriptions, completing partially written code, infilling code at the cursor position, debugging, and explaining existing code.[2] Because all of its weights are publicly downloadable, developers can run the models locally, [fine-tune](/wiki/fine_tuning) them for domain-specific work, and embed them in their own tooling without depending on closed API services.[1] Meta framed the public release around developer access, stating that "publicly available, code-specific models can facilitate the development of new technologies that improve peoples' lives."[2]

## Background and Motivation

Before Code Llama, several models had demonstrated the potential of language models trained on code. OpenAI's [Codex](/wiki/openai_codex), the model behind [GitHub Copilot](/wiki/github_copilot), showed that [GPT](/wiki/gpt4)-based architectures could be adapted for code generation.[7] The open-source community produced models like StarCoder (developed by BigCode and Hugging Face) and CodeGen (from Salesforce), which offered publicly available alternatives but with certain limitations in performance or licensing terms.[6]

Meta's motivation for Code Llama was to provide a fully open-weight, commercially usable family of code models that could match or exceed the performance of proprietary systems.[2] By starting from the already capable Llama 2 foundation and applying targeted code specialization, the research team aimed to create models that developers could run locally, [fine-tune](/wiki/fine_tuning) for domain-specific tasks, and integrate into their own tools without reliance on closed API services.[1]

## Architecture

Code Llama inherits the [transformer](/wiki/attention) architecture of Llama 2, which is a decoder-only autoregressive transformer.[4] The architecture incorporates several modern design choices that distinguish it from the original transformer formulation:

- **Pre-normalization with RMSNorm:** Rather than applying layer normalization after each sub-layer (post-norm), Llama 2 normalizes the input to each sub-layer using Root Mean Square Layer Normalization (RMSNorm), which improves training stability.[4]
- **SwiGLU activation function:** The feed-forward network uses the SwiGLU activation function instead of the standard [ReLU](/wiki/relu), providing improved training performance as demonstrated by Shazeer (2020).[4]
- **Rotary Position Embeddings (RoPE):** Instead of absolute or learned positional [embeddings](/wiki/embeddings), the model uses Rotary Position Embeddings, which encode relative position information directly into the attention mechanism. This approach allows the model to generalize to sequence lengths not seen during training, a property that Code Llama exploits during long-context fine-tuning.[1]
- **Grouped-Query [Attention](/wiki/attention) (GQA):** The 34B and 70B models use grouped-query attention, which reduces the number of key-value heads relative to query heads. This decreases memory bandwidth requirements during inference and makes the KV cache more efficient, enabling faster generation with large models.[1]

The tokenizer is a byte-pair encoding (BPE) tokenizer shared with Llama 2, featuring a vocabulary of 32,000 tokens.[4]

| Parameter | 7B | 13B | 34B | 70B |
|---|---|---|---|---|
| Transformer layers | 32 | 40 | 48 | 80 |
| Attention heads | 32 | 40 | 48 | 64 |
| Embedding dimension | 4,096 | 5,120 | 8,192 | 8,192 |
| Context length (training) | 16,384 | 16,384 | 16,384 | 16,384 |
| Context length (inference) | 100,000 | 100,000 | 100,000 | 100,000 |
| Vocabulary size | 32,000 | 32,000 | 32,000 | 32,000 |
| GQA | No | No | Yes | Yes |

## Model Variants

Code Llama is offered in three distinct variants, each tailored to different use cases:

### Code Llama (Base)

The foundation model serves as the general-purpose code generation backbone. It is initialized from Llama 2 and further trained on a large corpus of code and code-adjacent data.[1] The base model supports a wide range of programming languages and is suitable for code completion, code generation from natural language descriptions, and general code understanding tasks.

### Code Llama-Python

This variant undergoes additional training on a Python-specific dataset of 100 billion tokens after the initial code training phase.[1] The Python-focused dataset comprises 75% Python code, 10% code in other languages, 10% natural language related to code, and 5% general natural language.[1] Code Llama-Python consistently outperforms the base variant on Python-specific benchmarks such as [HumanEval](/wiki/humaneval) and is designed for developers working primarily in Python.[1]

### Code Llama-Instruct

The instruction-tuned variant is fine-tuned to follow natural language instructions for programming tasks. It is designed for conversational coding assistance, where a user describes a task in plain English and the model generates the corresponding code. The instruction tuning uses a combination of proprietary instruction data and a self-instruct dataset generated through an automated pipeline (described in the Training section below).[1] Code Llama-Instruct is the recommended variant for interactive and safety-sensitive applications.[2]

| Variant | Description | Best For | FIM Support (7B/13B) | FIM Support (34B) | FIM Support (70B) |
|---|---|---|---|---|---|
| Code Llama (Base) | General-purpose code model | Code completion, generation | Yes | No | Yes |
| Code Llama-Python | Python-specialized | Python development | No | No | No |
| Code Llama-Instruct | Instruction-following | Conversational coding, Q&A | Yes | No | Yes |

## Training Process

The training of Code Llama follows a multi-stage pipeline, where each stage progressively adds capabilities to the base Llama 2 model. The entire training process, covering all 12 models (three variants at four sizes), required approximately 1.4 million GPU hours on NVIDIA A100-80GB hardware (each unit rated at a thermal design power of 350-400W), performed on Meta's Research Super Cluster.[1]

### Stage 1: Code Training

Starting from the pre-trained Llama 2 checkpoints, the models are further trained on a near-deduplicated dataset of publicly available code. The 7B, 13B, and 34B models were each trained on 500 billion tokens, while the 70B model (released later, in January 2024) was trained on 1 trillion tokens.[1][5]

The training data composition for this stage is:

| Data Source | Proportion |
|---|---|
| Open-source code from GitHub | 85% |
| Natural language about code (e.g., Stack Overflow discussions, code documentation) | 8% |
| General natural language | 7% |

The inclusion of natural language data, both code-related and general, is a deliberate design choice. It helps the model retain the strong natural language understanding capabilities of Llama 2, which is important for tasks like generating code from English descriptions or understanding code comments.[1]

The model is trained on code from multiple programming languages. The primary languages supported include Python, C++, Java, PHP, TypeScript (JavaScript), C#, and Bash.[1]

### Stage 2: Fill-in-the-Middle (FIM) Training

A key capability of Code Llama is its ability to perform code infilling, where the model generates code that fits between a given prefix and suffix. This is trained using a fill-in-the-middle (FIM) objective, applied to the 7B, 13B, and 70B models (but not the 34B model).[1]

During FIM training, a document is randomly split at two points into three segments: a prefix, a middle section, and a suffix. The model is then trained in a multitask fashion with two formats:

- **Prefix-Suffix-Middle (PSM):** The model receives the prefix, then the suffix, and must predict the middle section.[1]
- **Suffix-Prefix-Middle (SPM):** The model receives the suffix first, then the prefix, and must predict the middle section.[1]

The FIM transformation is applied with approximately 90% probability on training documents.[1] Importantly, the authors demonstrated that FIM training does not degrade standard left-to-right autoregressive performance. Models trained with FIM achieve comparable scores to models trained without it on standard benchmarks, while gaining the additional infilling capability.[1]

The 34B model was trained without the FIM objective. This decision was based on the observation that the 34B model was intended primarily for large-scale code generation tasks where infilling was considered less critical. However, the 70B model, released later, was trained with FIM support, responding to community demand for infilling in larger models.[5]

### Stage 3: Long Context Fine-Tuning (LCFT)

All Code Llama models undergo a dedicated long context fine-tuning stage that extends their effective context window. While Llama 2 was pre-trained with a context length of 4,096 tokens, Code Llama extends this to 16,384 tokens during training and demonstrates stable performance on sequences of up to 100,000 tokens at inference time.[1]

The key technique enabling this extension is a modification to the Rotary Position Embeddings. The base period of the RoPE frequency is increased from 10,000 (the Llama 2 default) to 1,000,000. This rescaling allows the positional encoding to represent much longer sequences without the periodic aliasing that would occur with the original frequency base. The long context fine-tuning phase uses a relatively small number of additional training steps, making it an efficient adaptation.[1]

The extended context capability is particularly valuable for code-related tasks, where developers frequently need to work with files containing thousands of lines. Repository-level code understanding, long function implementations, and cross-file dependency analysis all benefit from the larger context window.

### Stage 4: Instruction Fine-Tuning

The Code Llama-Instruct variants undergo an additional fine-tuning stage to learn how to follow natural language instructions. The instruction-tuning dataset is composed of two main sources:

1. **Proprietary instruction data:** A curated set of instruction-response pairs designed to improve safety and helpfulness.[1]
2. **Self-instruct dataset:** A machine-generated dataset created through an automated pipeline:
   - Llama 2 70B was prompted to generate approximately 62,000 interview-style programming questions.[1]
   - After deduplication, approximately 52,000 unique questions remained.[1]
   - Code Llama 7B was then used to generate unit tests and ten candidate solutions for each question.[1]
   - Solutions were validated by executing the generated unit tests, retaining only those that passed.[1]
   - The final dataset contained approximately 14,000 question-test-solution triplets.[1]

This self-instruct approach uses execution feedback as a quality signal rather than relying on expensive human annotation or [reinforcement learning from human feedback](/wiki/reinforcement_learning) ([RLHF](/wiki/rlhf)).[1] The instruction fine-tuning dataset also includes a rehearsal component: a small proportion of data from the original code training stage is mixed in to prevent catastrophic forgetting of the model's code generation capabilities.[1]

### Unnatural Code Llama

The paper also describes an experimental model called "Unnatural Code Llama," which is the Code Llama-Python 34B model fine-tuned on 15,000 samples from an "unnatural instructions" dataset (following the methodology of Honovich et al., 2023).[10] This model achieved notably strong results (reported as 62.2% on HumanEval and 61.2% on MBPP), demonstrating that even a small set of high-quality, diverse coding data can yield significant improvements. However, this model was not publicly released.[1]

## Fill-in-the-Middle (FIM) and Code Infilling

Code infilling is one of the most practically useful capabilities of Code Llama, directly applicable to integrated development environments (IDEs) and code editors. When a developer is writing code, the model can predict what should appear at the cursor position given the code that comes before and after the cursor.[1]

The infilling capability is supported in the following models:

| Model Size | Code Llama (Base) | Code Llama-Python | Code Llama-Instruct |
|---|---|---|---|
| 7B | Yes | No | Yes |
| 13B | Yes | No | Yes |
| 34B | No | No | No |
| 70B | Yes | No | Yes |

The FIM approach uses special tokens to delineate the prefix, middle, and suffix regions. During inference, a user provides the code before the cursor (prefix) and the code after the cursor (suffix), and the model generates the missing middle section.[1] This is distinct from standard autoregressive generation, which can only continue from the end of a given prompt.

Applications of code infilling include:

- **Code completion in editors:** Suggesting code at the cursor position within an existing file.
- **Docstring generation:** Given a function signature (prefix) and body (suffix), generating the documentation string in between.
- **Type annotation insertion:** Filling in type hints between function signatures and return statements.
- **Test case generation:** Inserting test logic between setup and assertion code.

## Benchmark Performance

Code Llama was evaluated on several widely used coding benchmarks. The primary benchmarks are HumanEval (a set of 164 hand-written Python programming problems by OpenAI) and [MBPP](/wiki/mbpp) (Mostly Basic Python Programming, containing 974 crowd-sourced Python tasks from Google).[7][9] Results are reported as pass@1, the percentage of problems solved correctly on the first attempt.[1]

### HumanEval Results (pass@1)

| Model | Size | HumanEval (pass@1) |
|---|---|---|
| Llama 2 | 7B | 12.2% |
| Llama 2 | 13B | 20.1% |
| Llama 2 | 70B | 30.5% |
| StarCoder Base | 15.5B | 30.4% |
| code-cushman-001 ([Codex](/wiki/openai_codex)) | 12B | 33.5% |
| Code Llama (Base) | 7B | 33.5% |
| Code Llama (Base) | 13B | 36.0% |
| Code Llama (Base) | 34B | 48.8% |
| Code Llama (Base) | 70B | 53.0% |
| Code Llama-Python | 7B | 38.4% |
| Code Llama-Python | 13B | 43.3% |
| Code Llama-Python | 34B | 53.7% |
| Code Llama-Python | 70B | 57.3% |
| Code Llama-Instruct | 7B | 34.8% |
| Code Llama-Instruct | 13B | 42.7% |
| Code Llama-Instruct | 34B | 41.5% |
| Code Llama-Instruct | 70B | 67.8% |
| GPT-3.5 (ChatGPT) | N/A | 48.1% |
| [GPT-4](/wiki/gpt4) | N/A | 67.0% |

A notable finding is that Code Llama-Python 7B (38.4%) outperforms Llama 2 70B (30.5%) on HumanEval, demonstrating the effectiveness of domain-specific code training.[1] At the largest scale, Code Llama-Instruct 70B (67.8%) slightly surpasses GPT-4 (67.0%) on the HumanEval benchmark in zero-shot evaluation.[1]

### MBPP Results (pass@1)

| Model | Size | MBPP (pass@1) |
|---|---|---|
| Llama 2 | 70B | 45.4% |
| StarCoder Base | 15.5B | 43.6% |
| code-cushman-001 (Codex) | 12B | 45.9% |
| Code Llama (Base) | 7B | 41.4% |
| Code Llama (Base) | 13B | 47.0% |
| Code Llama (Base) | 34B | 55.0% |
| Code Llama (Base) | 70B | 62.4% |
| Code Llama-Python | 7B | 47.6% |
| Code Llama-Python | 13B | 49.0% |
| Code Llama-Python | 34B | 56.2% |
| Code Llama-Python | 70B | 65.6% |
| Code Llama-Instruct | 7B | 44.4% |
| Code Llama-Instruct | 13B | 49.4% |
| Code Llama-Instruct | 34B | 57.0% |
| Code Llama-Instruct | 70B | 62.2% |
| GPT-3.5 (ChatGPT) | N/A | 52.2% |

### MultiPL-E Results

Code Llama was also evaluated on the MultiPL-E benchmark, which extends HumanEval to multiple programming languages.[8] The Code Llama 70B model demonstrated strong performance across languages:

| Language | Code Llama 70B (pass@1) |
|---|---|
| Python | 52.8% |
| C++ | 51.9% |
| Java | 50.9% |
| PHP | 43.5% |
| TypeScript | 51.3% |
| C# | 42.1% |
| Bash | 31.5% |

These results show that while Code Llama performs strongest in Python (especially the Python variant), it maintains competitive performance across a variety of programming languages.[1]

### Comparison with Other Models

The following table summarizes how Code Llama's best results compare with other prominent code generation models at the time of its release:

| Model | Organization | Parameters | HumanEval (pass@1) | MBPP (pass@1) | Open Weights |
|---|---|---|---|---|---|
| Code Llama-Instruct 70B | [Meta](/wiki/meta_ai) | 70B | 67.8% | 62.2% | Yes |
| GPT-4 | [OpenAI](/wiki/openai) | Undisclosed | 67.0% | N/A | No |
| GPT-3.5 (ChatGPT) | OpenAI | Undisclosed | 48.1% | 52.2% | No |
| Code Llama-Python 34B | Meta | 34B | 53.7% | 56.2% | Yes |
| StarCoder Base | BigCode / [Hugging Face](/wiki/hugging_face) | 15.5B | 30.4% | 43.6% | Yes |
| code-cushman-001 (Codex) | OpenAI | 12B | 33.5% | 45.9% | No |
| Llama 2 | Meta | 70B | 30.5% | 45.4% | Yes |

Code Llama-Instruct 70B was the first open-weight model to match GPT-4 on HumanEval, a significant milestone for the open-source AI community.[5]

## When was Code Llama released?

Code Llama was released in two phases. The initial release on August 24, 2023 covered the 7B, 13B, and 34B sizes (nine models across the three variants), and the 70B models followed on January 29, 2024.[1][5]

| Date | Release | Details |
|---|---|---|
| August 24, 2023 | Code Llama 7B, 13B, 34B | Initial release of all three variants (base, Python, Instruct) at 7B, 13B, and 34B sizes. Nine models total. |
| January 29, 2024 | Code Llama 70B | Release of the 70B parameter models in all three variants. Trained on 1 trillion tokens (double the smaller models). Added FIM support at 70B scale. |

The 70B release was notable because it was trained on twice the number of tokens as the earlier models (1 trillion vs. 500 billion) and introduced FIM infilling capability at the 70B scale, which was not available in the 34B model.[5]

## Is Code Llama open source?

Code Llama is released under the same community license as Llama 2, which permits both research and commercial use.[2][4] The key terms of the license include:

- **Free for research and commercial use:** Organizations and individuals can use, modify, and distribute the model weights and code.[2]
- **Attribution required:** Derivative works must include a copy of the license and attribution to Meta.[4]
- **Monthly active user threshold:** Companies with more than 700 million monthly active users in the preceding calendar month must request a separate license from Meta. This provision effectively requires the largest technology companies (such as Google, Amazon, or Apple) to negotiate directly with Meta before deploying the model.[4]
- **Use restriction:** The model cannot be used to train other AI models (LLM or otherwise) using the model's output, under the terms of the Llama 2 Community License.[4]
- **Acceptable Use Policy:** The license includes restrictions on using the model for harmful purposes, including generating malware, conducting cyberattacks, or creating weapons of mass destruction.[4]

It is worth noting that the Open Source Initiative has stated that the Llama 2 Community License does not meet the traditional definition of "open source" due to its commercial restrictions for very large companies and other usage limitations. Nonetheless, the license is substantially more permissive than those of comparable proprietary models like Codex or GPT-4.

## Community Adoption and Ecosystem

Code Llama has seen broad adoption across the developer community and has been integrated into numerous tools and platforms.

### Local Inference

One of Code Llama's key advantages is that developers can run the models locally without relying on cloud APIs. Several tools facilitate local deployment:

- **[Ollama](/wiki/ollama):** A popular open-source tool for running LLMs locally, with first-class support for all Code Llama variants. Users can download and run Code Llama models with a single command.
- **[llama.cpp](/wiki/llama_cpp):** The C/C++ inference engine supports Code Llama models with various quantization formats ([GGUF](/wiki/gguf)), enabling efficient inference on consumer hardware including Apple Silicon Macs.
- **[LM Studio](/wiki/lmstudio):** A desktop application that provides a graphical interface for downloading and running Code Llama models locally.

### IDE Integration

Code Llama has been integrated into code editors through several extensions and plugins:

- **Continue:** An open-source VS Code and JetBrains extension that connects editors with local LLMs, including Code Llama, for inline code completion and chat-based coding assistance.
- **CodeGPT:** A VS Code extension with over one million downloads that supports Code Llama through Ollama for local AI-powered coding.
- **Tabby:** An open-source, self-hosted AI coding assistant that supports Code Llama as a backend model.

### Fine-Tuned Derivatives

The open nature of Code Llama's weights has enabled the community to create numerous fine-tuned derivatives:

- **Phind-CodeLlama-34B:** Phind fine-tuned Code Llama 34B on their proprietary dataset of high-quality code, achieving 73.8% on HumanEval at the time of its release, significantly surpassing the base Code Llama-Python 34B score of 53.7%.
- **CodeFuse-CodeLlama-34B:** Created by the CodeFuse team at Ant Group, this model was fine-tuned for improved coding capabilities.
- **WizardCoder variants:** The WizardCoder project applied Evol-Instruct methods to Code Llama models, producing models with enhanced instruction-following for code.

### Cloud and API Services

Code Llama models are available through various cloud inference platforms, including [Amazon Web Services](/wiki/amazon_web_services) (via SageMaker and Bedrock), [NVIDIA](/wiki/nvidia) NIM, Hugging Face [Inference](/wiki/inference) Endpoints, [Perplexity](/wiki/perplexity), [Together AI](/wiki/together_ai), and [Replicate](/wiki/replicate), making them accessible to developers who prefer API-based access.

## How does Code Llama compare to other code models?

### StarCoder

[StarCoder](/wiki/starcoder) is a 15.5B parameter code generation model developed by the BigCode project, a collaboration between Hugging Face and ServiceNow.[6] StarCoder was trained on The Stack, a curated dataset of permissively licensed source code.[6] While StarCoder achieved strong results for its time (30.4% on HumanEval), Code Llama surpassed it across all model sizes. Even Code Llama 7B (33.5% on HumanEval) outperforms StarCoder Base despite having fewer than half the parameters.[1] StarCoder does offer a larger context window of 8,192 tokens natively, though Code Llama's long context fine-tuning extends its effective context far beyond this.[6]

### Codex

OpenAI's Codex (code-cushman-001, a 12B parameter model) was the model powering GitHub Copilot at launch.[7] Codex scored 33.5% on HumanEval, matching Code Llama 7B.[1] However, Codex was a proprietary model with no publicly available weights, while Code Llama provides comparable or superior performance in a fully downloadable, modifiable package. OpenAI deprecated the Codex API in March 2023, months before Code Llama's release.

### GPT-4

At the time of Code Llama's initial release in August 2023, GPT-4 scored 67.0% on HumanEval, well above any Code Llama model.[1] However, the January 2024 release of Code Llama 70B-Instruct closed this gap entirely, achieving 67.8% on HumanEval, a score slightly above GPT-4's.[5] It is important to note that GPT-4 is a much larger, general-purpose model, while Code Llama is specialized for code. Also, subsequent GPT-4 updates and the release of models like GPT-4o and o1 have since raised the bar for proprietary models on coding benchmarks.

### DeepSeek Coder

[DeepSeek](/wiki/deepseek) Coder, released in late 2023 by the Chinese AI lab DeepSeek, offered competitive performance with Code Llama across various benchmarks. DeepSeek Coder models ranged from 1.3B to 33B parameters and were trained from scratch on a 2-trillion-token code corpus. DeepSeek Coder 33B achieved around 56% on HumanEval, competitive with Code Llama-Python 34B's 53.7%.

## Limitations

Despite its strong performance, Code Llama has several limitations:

- **Not the latest generation:** As of 2025, Code Llama has been superseded by newer models, including Meta's own Llama 3 and Llama 3.1 families, which demonstrate stronger coding performance. The broader landscape has also shifted, with models like [Claude](/wiki/claude) 3.5 Sonnet, GPT-4o, and DeepSeek Coder V2 pushing coding benchmarks further.
- **34B infilling gap:** The 34B model does not support fill-in-the-middle infilling, which limits its usefulness for IDE-based code completion. Users requiring both large model capacity and infilling must use the 70B model, which has higher hardware requirements.[5]
- **Benchmark saturation:** The HumanEval and MBPP benchmarks, while widely used, contain relatively simple programming problems. Performance on these benchmarks may not fully reflect a model's ability to handle real-world software engineering tasks involving large codebases, complex dependencies, or multi-file changes.
- **Safety considerations:** While the Instruct variants include safety training, Code Llama may still generate insecure, buggy, or potentially harmful code. The model does not guarantee that generated code is free of vulnerabilities, and developers should always review and test generated code thoroughly.
- **Training data cutoff:** The model's knowledge is limited to data available before its training cutoff (early 2023 for most training data). It may not be aware of newer libraries, API changes, or language features released after this date.

## Significance and Impact

Code Llama represented a significant milestone in the democratization of AI-assisted coding. It was one of the first open-weight code models to match the performance of proprietary alternatives like GPT-4 on standard benchmarks.[1] The release of model weights, combined with the relatively permissive community license, enabled a wide ecosystem of tools, fine-tuned models, and applications that would not have been possible with closed models.

The model's influence extends beyond its direct use. Its training methodology, particularly the multi-stage pipeline of code specialization, FIM training, and long context fine-tuning, has informed subsequent work on code LLMs.[1] The self-instruct approach for generating coding instruction data without human annotation has been adopted and refined by other research groups.

Code Llama also demonstrated the viability of the "foundation model plus specialization" approach to building code models. Rather than training a code model from scratch, Meta showed that starting from a strong general-purpose language model and applying targeted code training could yield competitive results with lower total compute cost.[1]

## See Also

- [LLM Compiler (Meta)](/wiki/llm_compiler)
- [Llama](/wiki/llama)
- [Codex](/wiki/openai_codex)
- [GitHub Copilot](/wiki/github_copilot)
- [Fine-Tuning](/wiki/fine_tuning)
- [HumanEval](/wiki/humaneval)
- [MBPP](/wiki/mbpp)

## References

1. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Defossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., & Synnaeve, G. (2023). "Code Llama: Open Foundation Models for Code." arXiv preprint arXiv:2308.12950. https://arxiv.org/abs/2308.12950
2. Meta AI. (2023, August 24). "Introducing Code Llama, a state-of-the-art large language model for coding." Meta AI Blog. https://ai.meta.com/blog/code-llama-large-language-model-coding/
3. Meta AI. (2023, August 24). "Introducing Code Llama, an AI Tool for Coding." Meta Newsroom. https://about.fb.com/news/2023/08/code-llama-ai-for-coding/
4. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv preprint arXiv:2307.09288. https://arxiv.org/abs/2307.09288
5. Meta AI. (2024, January 29). "Code Llama 70B." Hugging Face. https://huggingface.co/codellama/CodeLlama-70b-hf
6. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., et al. (2023). "StarCoder: may the source be with you!" arXiv preprint arXiv:2305.06161. https://arxiv.org/abs/2305.06161
7. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P., Kaplan, J., et al. (2021). "Evaluating Large Language Models Trained on Code." arXiv preprint arXiv:2107.03374. https://arxiv.org/abs/2107.03374
8. Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., et al. (2023). "MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation." IEEE Transactions on Software Engineering, 49(7), 3675-3691.
9. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., et al. (2021). "Program Synthesis with Large Language Models." arXiv preprint arXiv:2108.07732. https://arxiv.org/abs/2108.07732
10. Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023). "Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor." ACL 2023. https://arxiv.org/abs/2212.09689