Fill-in-the-middle

Fill-in-the-middle (FIM) is a training objective and inference technique that lets an autoregressive language model generate text for a gap in the middle of a document, conditioned on both the text before the gap (the prefix) and the text after it (the suffix). The trick is a data transformation applied during pretraining: a span is cut out of the middle of a training document and moved to the end, with special tokens marking where the pieces came from. The model then learns, through ordinary next-token prediction, to produce the missing middle after reading the prefix and suffix. The name and the standard recipe come from "Efficient Training of Language Models to Fill in the Middle," a July 2022 paper by Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen at OpenAI ^[1].

FIM matters most for code. A developer's cursor usually sits in the middle of a file, between existing code above and below, so a completion model that only sees the prefix is working half blind. Nearly every major open code model since 2023 has trained with a FIM objective, including StarCoder, Code Llama, DeepSeek-Coder, CodeGemma, Qwen2.5-Coder, and Codestral ^{[6][7][8][9][10][12]}, and GitHub Copilot credits FIM prompting with a measurable jump in accepted completions ^[11].

Background

A standard decoder-only language model factorizes text left to right: each token is predicted from the tokens before it. That makes generation past the end of a prompt natural, but it gives the model no way to look at text that comes after the insertion point. Infilling, by contrast, requires conditioning on both sides of a gap.

Earlier objectives handled two-sided context in other ways. BERT-style masked language modeling predicts masked tokens using a bidirectional encoder, but it fills short masked positions rather than generating open-ended spans, and the encoder is not built for free-form sampling. T5 introduced a span-corruption denoising objective for encoder-decoder models, in which random spans are dropped from the input and the decoder reconstructs them in order, marked by sentinel tokens ^[5]. Donahue, Lee, and Liang's 2020 "Enabling Language Models to Fill in the Blanks" showed that ordinary left-to-right models could be trained or fine-tuned on "sequences containing the concatenation of artificially-masked text and the text which was masked," so that infilling becomes a plain language modeling task ^[3]. For code specifically, InCoder (Fried et al., April 2022) trained a generative model with a causal masking objective that moves masked regions of a file to its end, motivated by the observation that "code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined" ^[4].

The OpenAI FIM paper, released a few months after InCoder, turned this family of ideas into a measured pretraining recipe: it ran the transformation at scale in decoder-only pretraining, ablated the key hyperparameters, and quantified what the extra capability costs ^[1].

How the transformation works

During data preparation, a document (or a packed training context) is split at two random positions into three pieces: prefix, middle, and suffix. Bavarian et al. select the split points uniformly at random at the character level, before tokenization, so each piece is about a third of the document in expectation ^[1]. The pieces are then reassembled with sentinel tokens in one of two orders:

Scroll sideways for more →

The SPM layout shown is the variant the paper actually trains on. A more obvious encoding, <SUF> suffix <PRE> prefix <MID> middle, is discussed in the paper's appendix, but the authors keep the <PRE> sentinel first because that form arises naturally inside PSM data (whenever the sampled prefix is empty) and so maximizes transfer between the two formats when they are trained jointly ^[1].

Training on the reordered sequence is ordinary next-token prediction; nothing about the architecture or the loss changes. The model simply learns that after a <MID> sentinel, it should produce text that connects the prefix to the suffix, and then emit an end-of-span token. At inference time, a completion system places the code before the cursor into the prefix slot and the code after the cursor into the suffix slot, samples the middle, and splices the result back into the file.

The paper's ablations produced several durable defaults ^[1]:

• The FIM rate (the fraction of training data given the transformation) can be high. Any value between 50% and 90% is described as a reasonable choice, and rates up to 90% showed no cost to left-to-right capability.
• Applying the transformation after documents are packed into training contexts (context-level FIM) consistently beat applying it to whole documents beforehand (document-level FIM).
• Training jointly on PSM and SPM with a 50-50 split worked best, and character-level random spans proved more robust than line-based spans: models trained only on line-based middles did slightly better on line-oriented benchmarks but, in the paper's words, failed "almost completely in the random span infilling benchmark" ^[1].
• SPM has a deployment advantage for live completion: "with SPM, appending tokens to the prefix no longer invalidates the keys and values computed in the suffix section," so a growing prefix (a user typing) does not force recomputation of the cached suffix. The authors note the benefit depends on the application, since an edited suffix invalidates the prefix cache instead ^[1].

The FIM-for-free result

The paper's central empirical claim, which it calls the FIM-for-free property, is that adding this objective costs essentially nothing. Training with a large fraction of transformed data "does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales" ^[1]. The authors trained eight model sizes from 50M to 6.9B parameters, on both code and natural language, for 100B tokens each, and found the property held not just at the final checkpoint but throughout training ^[1].

The comparison with fine-tuning is the sharper half of the result. Teaching FIM to an already-pretrained model required a large amount of additional compute relative to pretraining: in the paper's sweep of 16 fine-tuning runs, only the single most aggressive configuration (a 90% FIM rate, the full pretraining learning rate, and 50B tokens of fine-tuning) caught up to a baseline that had learned FIM during pretraining, and every milder setting fell short ^[1]. On that basis the authors suggested that "future autoregressive language models be trained with FIM by default" ^[1], a recommendation the code-model ecosystem largely followed. OpenAI released an infilling model trained with the paper's best practices in its API, along with a suite of infilling benchmarks derived from HumanEval (single-line, multi-line, random-span, and random-span-light splits) published in the openai/human-eval-infilling repository ^[1][2].

Adoption in code models

FIM became a standard ingredient of code-model pretraining within a year of the paper. The specifics vary by family:

Scroll sideways for more →

StarCoder, a 15.5B-parameter model with an 8K context window trained on roughly one trillion tokens of permissively licensed GitHub code from The Stack, applied the FIM transformation with an explicit citation to Bavarian et al., following design choices from the earlier SantaCoder work ^[6]. Code Llama, derived from Llama 2, extended its tokenizer with four special tokens for the prefix, middle, suffix, and end of the infilled span ^[7]. DeepSeek-Coder's ablations are useful reading on the trade-offs: a 100% FIM rate peaked on infilling benchmarks but produced the weakest ordinary code completion, and a 50% PSM rate outperformed a masked-span-prediction (MSP) variant, which is why the team settled on 50% PSM ^[8]. CodeGemma describes its recipe as FIM "with improvements that address the shortcomings cited in the original work as well as empirically-found systemic issues with existing FIM-trained models" ^[9].

One practical consequence of this diversity: the sentinel tokens and expected format differ across families. DeepSeek-Coder's three sentinels are not interchangeable with StarCoder's four or CodeGemma's <|fim_prefix|> family, so editors and completion servers must build the prompt with each model's exact template and mode (PSM or SPM) for infilling to work ^[6][8][9].

Editor completion

Inline completion in an editor is an infilling problem, and FIM is how production systems handle it. GitHub described the change in Copilot in May 2023: where earlier versions prompted the model only with code before the cursor, "with FIM, we can tell the model which part of the prompt is the prefix, and which part is the suffix," letting the model account for the code the developer has already written below the insertion point. GitHub reported that "FIM gave a 10% relative boost in performance, meaning developers accepted 10% more of the completions that were shown to them," with prompt caching keeping the added context from increasing latency ^[11].

Vendors of completion-focused models make the same argument from the supply side. Announcing Codestral 25.01, Mistral AI called the model state of the art "for FIM use cases across the board," and quoted Ty Dunn, co-founder of the Continue coding assistant: "For AI code assistants, code completion constitutes a large portion of the work, which requires models that are great at fill-in-the-middle (FIM)" ^[12]. The volume involved is large: Cursor reported in September 2025 that its Tab completion model "runs on every user action, handling over 400 million requests per day," and that an online reinforcement learning pipeline raised its accept rate by 28% while cutting the number of suggestions shown by 21% ^[13].

Relation to span corruption

FIM is best understood as the decoder-only relative of T5's span corruption. Both delete spans from a document and ask the model to reconstruct them with sentinel markers; the difference is architectural. Span corruption feeds the corrupted text to an encoder and reconstructs the missing spans with a separate decoder, which suits fine-tuning-based transfer but complicates open-ended sampling ^[5]. FIM keeps a single decoder and folds the reconstruction into the same left-to-right stream used for everything else, so one model serves both plain code generation and infilling without an architecture change ^[1]. Masked language modeling sits further away still: it recovers individual masked positions with a bidirectional encoder and is used mainly for representation learning rather than generation.

The empirical comparisons that exist favor the moved-span formulation for generative infilling. DeepSeek-Coder's ablation found its 50% PSM configuration outperformed an MSP (masked span prediction) strategy of the same rate on infilling evaluation ^[8].

Limitations

The capability is not literally free in every respect. DeepSeek-Coder documented a trade-off between infilling skill and ordinary completion as the FIM rate climbs toward 100% ^[8], which is why production recipes sit in the 50-90% band rather than at the maximum. Span-selection policy leaks into downstream behavior: models trained on tidy line-based spans degrade badly when asked to fill arbitrary character ranges ^[1]. For models that skipped FIM during pretraining, retrofitting it by fine-tuning is possible but compute-hungry relative to having trained with it from the start ^[1]. And because every family ships its own sentinel vocabulary and preferred mode, infilling support in tooling remains a per-model integration rather than a universal interface ^[6][8][9].

See also

• AI code generation
• GitHub Copilot
• StarCoder
• Code Llama
• Masked language model
• T5

References

1. ^Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., Chen, M. "Efficient Training of Language Models to Fill in the Middle." arXiv, July 28, 2022. arxiv.org/...2207.14255
2. ^OpenAI. "human-eval-infilling: evaluation harness for the HumanEval infilling benchmarks described in the FIM paper." GitHub (archived May 29, 2026). github.com/...human-eval-infilling
3. ^Donahue, C., Lee, M., Liang, P. "Enabling Language Models to Fill in the Blanks." arXiv, May 11, 2020. arxiv.org/...2005.05339
4. ^Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., et al. "InCoder: A Generative Model for Code Infilling and Synthesis." arXiv, April 12, 2022. arxiv.org/...2204.05999
5. ^Raffel, C., Shazeer, N., Roberts, A., et al. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." arXiv, October 23, 2019. arxiv.org/...1910.10683
6. ^Li, R., Ben Allal, L., Zi, Y., Muennighoff, N., Kocetkov, D., et al. "StarCoder: may the source be with you!" arXiv, May 9, 2023. arxiv.org/...2305.06161
7. ^Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., et al. "Code Llama: Open Foundation Models for Code." arXiv, August 24, 2023. arxiv.org/...2308.12950
8. ^Guo, D., Zhu, Q., Yang, D., et al. "DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence." arXiv, January 25, 2024. arxiv.org/...2401.14196
9. ^CodeGemma Team, Google. "CodeGemma: Open Code Models Based on Gemma." arXiv, June 17, 2024. arxiv.org/...2406.11409
10. ^Qwen Team. "Qwen2.5-Coder Technical Report." arXiv, September 18, 2024. arxiv.org/...2409.12186
11. ^Rosenkilde, J. "How GitHub Copilot is getting better at understanding your code." GitHub Blog, May 17, 2023. github.blog/...g-better-at-understanding-your-code
12. ^Mistral AI. "Codestral 25.01." January 13, 2025. mistral.ai/...codestral-2501
13. ^Cursor. "Improving Cursor Tab with online RL." September 12, 2025. cursor.com/...tab-rl