Last reviewed
Sources
56 citations
Review status
Source-backed
Revision
v5 · 6,662 words
Add missing citations, update stale details, or suggest a clearer explanation.
A context window (also called context length) is the maximum number of tokens that a large language model (LLM) can process at once, spanning both the input prompt and the generated output. It functions as the model's "working memory": the span of text, code, audio, or video the model can attend to when generating a response.[1][2] When input exceeds this limit, the earliest information is truncated and effectively "forgotten" by the model.[1]
Since GPT-1 launched with 512 tokens in 2018, context windows have grown to over 10 million tokens in specialized models such as Meta's Llama 4 Scout (released April 5, 2025), a roughly 20,000-fold increase in seven years that fundamentally transformed the capabilities of artificial intelligence (AI) systems.[3] This rapid expansion has unlocked applications previously impossible, from analyzing entire legal case files to understanding massive codebases and synthesizing extensive research literature in a single session.
A context window functions as a large language model's working memory: the maximum number of tokens the model can process simultaneously. Unlike human memory, which can selectively recall information from across a lifetime, transformer-based models operate on a fixed window: when input exceeds this limit, older information must be truncated or the model fails to process the request.[1]
The context window encompasses both input and output tokens. For instance, OpenAI's GPT-4o has a 128,000-token context window but limits output generation to 16,384 tokens, meaning the total conversation history plus the model's response cannot exceed 128,000 tokens.[4] This distinction between total context capacity and maximum output length has important practical implications for application design.
LLMs process text using fixed-size context windows with a sliding window approach. As new tokens are fed in (for example, as a conversation progresses), older tokens eventually fall outside the window and are no longer consulted by the model's attention mechanism.[5] This can cause models to lose track of earlier details; a chatbot may go off-topic or contradict something from earlier once that information slips out of context.
Smaller context windows lead models to "forget" the content of recent conversations, which can cause them to go off topic. After a few thousand words, they may forget their initial instructions, leading to responses based only on the last information in their context window.[6]
Tokens form the fundamental units that models process, and understanding tokenization is essential to grasping context window limitations. A token does not equal a word; it represents the smallest meaningful unit a model recognizes, which can be a complete word, a subword fragment, punctuation, or even a single character.[7] In English, the approximate ratio is 1.5 tokens per word, so "Hello, world!" might consume 3-4 tokens rather than 2.
Byte-Pair Encoding (BPE), originally a compression algorithm adapted for NLP by Sennrich et al. in 2016, forms the backbone of models like GPT-2, GPT-3, and GPT-4.[8] BPE begins with individual characters as its vocabulary and iteratively merges the most frequent adjacent character pairs into new tokens until reaching a predefined vocabulary size (typically 32,000-50,000 tokens). This approach balances vocabulary size against the ability to represent any text, including rare words or neologisms.
Byte-Level BPE, used by GPT-2 and successors, operates on raw UTF-8 bytes rather than characters, guaranteeing that any Unicode character can be represented using just 256 base tokens. This innovation eliminates "unknown" tokens entirely while maintaining a manageable vocabulary size.[8]
WordPiece, developed by Google and used in BERT and related models, employs a likelihood-based merging criterion. Rather than simply choosing the most frequent pair, WordPiece selects merges that maximize the probability of the training data, often producing more semantically meaningful subword units.[9]
Unigram, implemented in models like XLNet and ALBERT, reverses the BPE approach. It starts with a massive vocabulary of all possible substrings and progressively removes tokens that contribute least to the likelihood of the training corpus, ultimately retaining only the most informative units.[8]
SentencePiece, a language-agnostic tokenizer used in T5, XLNet, and modern multilingual models, treats input as a raw stream of Unicode characters (including spaces) rather than pre-tokenized words.[10] This design proves critical for languages without explicit word boundaries, such as Chinese, Japanese, and Thai, where traditional tokenization fails.
Tokenization efficiency varies dramatically across languages, creating significant inequities in context window utilization. Research demonstrates that the same sentence translated into Telugu requires approximately 7 times more tokens than its English equivalent despite having fewer characters.[1] English speakers can effectively fit 2-3 times more semantic content into the same context window compared to speakers of many Asian and African languages. This bias stems from training data composition, as models trained predominantly on English text develop tokenizers optimized for English morphology and character patterns.
At the heart of context windows lies the self-attention mechanism, introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al.[11] This mechanism fundamentally changed how neural networks process sequential data. The paper had accumulated more than 200,000 citations on Google Scholar as of December 2025, placing it among the most-cited works in the history of artificial intelligence.[11]
Self-attention allows each token to weigh the relevance of every other token in the sequence. For each token, the model computes three vectors through learned linear transformations:
Query (Q): Represents what the current token is looking for
Key (K): Represents what each token offers
Value (V): Represents the actual information each token contains
The attention output is computed as:[12]
Attention(Q, K, V) = softmax(QK^T / √d_k) × V
Where d_k is the dimension of the key vectors. The scaling factor √d_k prevents the dot products from growing too large, which would push the softmax function into regions with extremely small gradients, hampering learning.
Rather than performing attention once, transformer models split the Q, K, and V matrices into multiple "heads" (typically 8-32), each learning different relationships.[5] For example, one attention head might focus on syntactic dependencies while another captures semantic relationships or long-range coreference. The outputs of all heads are concatenated and projected through a final linear layer.
Original transformer specifications used 512-dimensional embeddings split across 8 heads, giving each head 64 dimensions. Modern large language models often use far larger dimensions (4096-8192) with proportionally more heads.
The most fundamental limitation of context windows stems from self-attention's O(n²) computational complexity with respect to sequence length n.[13] When computing attention, each token must attend to every other token, creating an n×n attention matrix. Doubling the sequence length quadruples memory requirements and computational cost.
For a concrete example: processing 1,000 tokens requires 1 million attention computations, while processing 10,000 tokens requires 100 million, a 100-fold increase for a 10-fold increase in context. This quadratic scaling creates severe bottlenecks as context windows expand.
Google researchers (Duman Keles et al., 2023) proved that self-attention time complexity is necessarily quadratic unless the Strong Exponential Time Hypothesis (SETH) is false, a fundamental result in computational complexity theory.[13] This means no algorithmic trick can reduce attention to linear time without sacrificing the full expressivity of standard attention or making implausible assumptions about computational hardness.
The memory complexity presents an even more pressing constraint. Modern GPUs are memory-bandwidth-bound rather than compute-bound, meaning data transfer between GPU memory levels (SRAM, HBM, system RAM) becomes the bottleneck.[14] Standard attention constantly moves the large n×n attention matrix between fast on-chip SRAM (~20MB capacity) and slower High Bandwidth Memory (~40-80GB), creating severe inefficiencies.
For large-scale models processing long sequences, the KV cache (key-value pairs stored for each token) requires substantial memory. Modern production systems typically require approximately 1MB per token of GPU memory for the KV cache alone, excluding model parameters and activation memory.[15]
When context windows extend to 128,000 tokens, memory requirements increase dramatically. Without optimizations, moving from a 4,000-token to a 128,000-token context window increases memory needs substantially, approaching the square of the length increase due to the quadratic attention complexity.[12]
| Year | Model | Context Window | Notes |
|---|---|---|---|
| 2018 | GPT-1 | 512 tokens | First GPT model |
| 2018 | BERT | 512 tokens | Bidirectional encoder |
| 2019 | GPT-2 | 1,024 tokens | Doubled context |
| 2020 | GPT-3 | 2,048 tokens | 175B parameters |
| 2022 | ChatGPT (GPT-3.5) | 4,096 tokens | Initial release |
| 2023 | GPT-3.5-Turbo-16K | 16,384 tokens | 4x increase |
| 2023 | GPT-4 | 8,192 / 32,768 tokens | Two variants |
| 2023 | Claude 2 | 100,000 tokens | First 100K model |
| 2023 | Claude 2.1 | 200,000 tokens | Industry milestone |
| 2023 | GPT-4 Turbo | 128,000 tokens | Major expansion |
| 2024 | Gemini 1.5 Pro | 1,000,000 tokens | Million-token milestone |
| 2024 | Gemini 1.5 Pro | 2,000,000 tokens | Extended further |
| 2024 | Llama 3 | 8,192 tokens | Open source |
| 2024 | Llama 3.1 | 128,000 tokens | 16x increase |
| 2025 | GPT-4.1 | ~1,000,000 tokens | API release |
| 2025 | GPT-5 | 400,000 tokens (API) | 272K input + 128K output |
| 2025 | Claude Sonnet 4.5 | 200,000 / 1,000,000 tokens | Beta extended context |
| 2025 | Llama 4 Scout | 10,000,000 tokens | Open source record |
| 2025 | Gemini 2.5 Pro | ~1,000,000 tokens | Current generation |
| 2026 | Gemini 3.1 Pro | 1,048,576 tokens | GA release Feb 2026 |
| 2026 | Claude Opus 4.6 / Sonnet 4.6 | 1,000,000 tokens | 1M context GA |
GPT-1 (June 2018), OpenAI's first Generative Pre-trained Transformer, launched with a modest 512-token context window (roughly 350-400 words or about one page of text).[16] This marked the beginning of the transformer's dominance in language modeling, though the limited context constrained practical applications.
BERT (October 2018), Google's bidirectional encoder model, maintained the same 512-token limit but processed context differently.[17] Unlike GPT's left-to-right generation, BERT considered both preceding and following context simultaneously, proving highly effective for classification and information extraction tasks within its limited window.
GPT-2 (February 2019) doubled the context to 1,024 tokens, a seemingly modest improvement that expanded the model's ability to maintain narrative coherence. With 1.5 billion parameters, GPT-2 demonstrated unprecedented text generation quality.
GPT-3 (June 11, 2020) doubled context again to 2,048 tokens (approximately 1,500 words or 3-4 pages).[16] With 175 billion parameters, GPT-3 demonstrated remarkable few-shot learning capabilities, establishing new benchmarks for in-context learning.
GPT-3.5-Turbo (November 2022), the model powering ChatGPT's initial release, featured a 4,096-token context window, enabling conversations spanning approximately 3,000 words.[1] This represented a significant milestone as it brought conversational AI to mainstream audiences.
GPT-3.5-Turbo-16K (June 2023) achieved a 16,384-token context window (equivalent to roughly 20 pages of text).[18] This expansion enabled processing of much longer documents and extended conversations.
GPT-4 (March 14, 2023) launched with two variants: an 8,192-token standard model and a 32,768-token extended version (GPT-4-32k).[19] The 32K version could handle about 25,000 words of text (around 50 pages), enabling use cases such as analyzing lengthy documents.
Claude 2 (July 2023), from Anthropic, achieved a breakthrough 100,000-token context window (roughly 75,000 words or 300 pages).[20] This represented an 11x increase over Claude 1's 9,000-token window and marked the first time a production model could process entire books. Anthropic demonstrated this capability by having Claude read the full text of The Great Gatsby (~72K tokens) and identify a single altered sentence, which it successfully did in under 22 seconds.
Claude 2.1 (November 21, 2023) doubled this achievement to 200,000 tokens (150,000 words or 500+ pages).[21] This capacity allowed Claude to digest and explain dense financial reports, compare themes across legal contracts, or sift through thousands of lines of code for debugging assistance.
GPT-4 Turbo (November 6, 2023) countered with a 128,000-token context window (equivalent to roughly 300 pages of text or 96,000 words).[22] This represented OpenAI's most significant context expansion to date.
Gemini 1.5 Pro (February 15, 2024) shattered previous records with a 1,000,000-token context window, a 5x leap capable of processing approximately 700,000 words, 11 hours of audio, or 1 hour of video.[23] The model utilized a Mixture-of-Experts (MoE) architecture, activating only a subset of parameters per input token, which helped manage computational costs.
Gemini 1.5 Pro - 2 Million Tokens (June 27, 2024) extended the context to 2,000,000 tokens (approximately 1.4 million words, 22 hours of audio, or 2 hours of video).[24] Google opened the 2M window to all developers in late June 2024, making it the largest commercially available context window at the time, and simultaneously launched context caching to reduce costs for repeated long prompts.[24]
Meta's Llama Family progressed rapidly:
Llama 2 (July 2023): 4,096 tokens
Llama 3 (April 2024): 8,192 tokens
Llama 3.1 (July 2024): 128,000 tokens (a 16x jump)[25]
Mistral AI's Evolution:
Mistral 7B (September 2023): 4,096 tokens with sliding window attention
Mistral Large 2 (July 2024): 128,000 tokens with 123 billion parameters[26]
Cohere's Command Models (2024) featured Command R and Command R+, both with 128,000-token context windows optimized for retrieval-augmented generation (RAG).[27]
GPT-4.1 (April 2025) features approximately 1 million tokens of context, described officially as supporting "up to 1 million tokens of context."[28] All three models in the GPT-4.1 family (GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano) share this context window capacity. The models show significant improvements in coding performance, scoring 54.6% on SWE-bench Verified, and demonstrate enhanced long-context comprehension capabilities.
GPT-5 (released August 7, 2025) is OpenAI's unified flagship model. Its API version accepts 272,000 input tokens and returns up to 128,000 output tokens, for a documented total context window of 400,000 tokens.[29][30] Inside ChatGPT, usable context is tiered by plan: 8,000 tokens for free users, 32,000 for Plus, and 128,000 for Pro, with the full 400,000-token window available only through direct API calls.[30] GPT-5 unifies advanced reasoning with general-purpose capabilities and features a real-time router that automatically determines how much "thinking" each query requires.[29]
Claude Sonnet 4.5 (September 2025) offers 200,000 tokens standard with 1,000,000 tokens available via the beta header context-1m-2025-08-07 on the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI.[31] The model achieves 77.2% on SWE-bench Verified, demonstrating strong software engineering capabilities. Claude Opus 4.1 maintains 200,000 tokens with 64,000-token output maximum.
Meta Llama 4 (April 5, 2025) achieved the most dramatic expansion among open-weight models:[3]
Llama 4 Scout: 10,000,000 tokens (the longest context window for an open-source model), built on a Mixture-of-Experts design with 109B total parameters (17B active across 16 experts) and a new positional architecture Meta calls iRoPE, fitting on a single Nvidia H100 GPU[3]
Llama 4 Maverick: 1,000,000 tokens with 400B parameters
Specialized Ultra-Long Context Models:
MiniMax-Text-01 (January 2025): 4,000,000 tokens with 456B parameters[32]
Magic.dev LTM-2-Mini (August 2024): 100,000,000 tokens, the longest context window publicly announced. Magic described it as "10 million lines of code, or 750 novels," and reported that for each decoded token its sequence-dimension algorithm is roughly 1000x cheaper than the attention mechanism in Llama 3.1 405B at a 100M-token context.[33]
Flash Attention (Dao et al., 2022) fundamentally transformed context window feasibility by addressing the memory bandwidth bottleneck.[14] The core insight: modern GPUs are memory-bound, not compute-bound. Standard attention constantly transfers the large n×n attention matrix between fast on-chip SRAM and slower High Bandwidth Memory, wasting time on data movement.
Flash Attention employs three innovations:
Tiling: Breaks computation into blocks that fit in SRAM
Fused Operations: Computes attention entirely in SRAM without writing intermediate results to HBM
Recomputation: Recalculates attention during the backward pass rather than storing intermediate values
Results on various sequence lengths:[14]
15% speedup on BERT-large (512 tokens)
3x speedup on GPT-2 (1K tokens)
59x memory reduction for inference at 16,384 tokens
Enables sequences up to 64K tokens on standard hardware
Flash Attention-2 (2023) improved parallelization, achieving 2x speedup over Flash Attention-1 through better GPU utilization and reduced synchronization overhead.[34]
Flash Attention-3 (2024), optimized for NVIDIA H100 GPUs, introduces warp specialization, pingpong scheduling, and incoherent processing.[35] Performance achievements:
1.5-2.0x faster than Flash Attention-2 with FP16
740 TFLOPS (75% of H100's theoretical maximum)
1.2 PFLOPS with FP8 precision
RoPE, introduced by Su et al. (2021), revolutionized how transformers handle positional information.[36] Unlike absolute positional encodings, RoPE encodes position through rotation matrices applied to query and key vectors.
For each position, RoPE rotates Q and K vectors in 2D planes with rotation angle proportional to position and inversely proportional to dimension. This design creates several advantages:
Sequence Length Flexibility: No fixed maximum length during training
Decaying Inter-token Dependency: Attention naturally decreases with distance
Parameter-Free: No additional parameters beyond standard attention
Relative Position Encoding: Captures relative distances between tokens
Adoption: RoPE powers Llama 3, Gemma, Qwen, Mistral, and most modern open-source models.[37]
Ring Attention (Liu et al., ICLR 2024) enables context windows to scale linearly with the number of available devices by distributing attention computation across GPUs in a ring topology.[38]
Mechanism:
Partition Q, K, V across N devices arranged in a ring
Each device computes local attention on its chunk
K and V blocks circulate around the ring
Each device processes N iterations, once per K/V block
Communication overlaps with computation
Scaling: Context length scales linearly with device count. With this approach, adding more GPUs directly increases the maximum context window the system can handle efficiently.
Striped Attention (2023) extends Ring Attention for causal transformers, providing better load balancing across devices.[39]
Sparse attention mechanisms reduce computational complexity by limiting which tokens attend to each other, trading some expressivity for efficiency.
BigBird (Google, NeurIPS 2020) combines three attention patterns to achieve O(n) complexity:[40]
Global Attention: Fixed tokens attend to all positions
Local Attention: Sliding window for immediate context
Random Attention: Sparse random connections for long-range dependencies
Longformer (Beltagy et al., 2020) pioneered sliding window attention with O(n×w) complexity, where each token attends to w/2 neighbors on each side.[41] This proved effective for document-level tasks while maintaining manageable computational costs.
| Mechanism | Key Idea | Computational Complexity | Primary Advantage |
|---|---|---|---|
| Full Attention (Baseline) | Every token attends to every other token | O(n²) | Maximum expressiveness; captures all dependencies |
| Sliding Window Attention | Each token attends to a fixed window of local neighbors | O(n × w) | High efficiency for local context tasks |
| Longformer | Combines sliding window attention with a few global tokens | O(n) | Balances local and global context for document-level tasks |
| BigBird | Combines sliding window, global, and random attention | O(n) | Universal approximator of full attention with theoretical guarantees |
| Reformer | Groups similar tokens using Locality-Sensitive Hashing (LSH) | O(n log n) | Extreme memory efficiency, suitable for very long sequences |
| Linformer | Projects Key and Value matrices to a lower dimension (low-rank approximation) | O(n) | Achieves linear complexity with strong theoretical foundation |
The Key-Value (KV) cache stores computed key and value vectors for previously processed tokens, enabling efficient autoregressive generation.[42] However, the KV cache can become a significant memory bottleneck for long contexts.
Instead of every query head having dedicated key/value heads, GQA shares K/V heads across multiple query heads:
Standard: 32 Q heads, 32 K heads, 32 V heads
GQA: 32 Q heads, 8 K heads, 8 V heads (4x reduction in KV cache)
MQA: 32 Q heads, 1 K head, 1 V head (32x reduction in KV cache)
Adoption: Llama 2, Llama 3, Mistral, and Falcon use GQA, reducing KV cache size by 4-8x with minimal quality degradation. This optimization is crucial for enabling longer context windows with constrained memory budgets.
Modern systems quantize KV cache to FP8 or INT8 precision, achieving 2-4x memory reduction. TensorRT-LLM and vLLM both support quantized caching in production deployments, allowing models to handle longer contexts with the same hardware.
ALiBi (Press et al., ICLR 2022) eliminates positional embeddings entirely, instead biasing attention scores with a simple linear penalty based on distance.[12] This remarkably simple approach enables:
Training on short sequences, inferring on longer sequences without fine-tuning
Reduced memory usage compared to sinusoidal positional embeddings
Better perplexity on long-document benchmarks like WikiText-103
The method applies a constant negative bias to attention scores proportional to the distance between tokens, naturally causing the model to pay more attention to nearby tokens.
Adoption: MPT (MosaicML), BLOOM (BigScience), Replit Code models, and Falcon variants successfully use ALiBi for position encoding.
The landmark paper "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023) revealed a critical limitation: even models with large context windows fail to robustly use information throughout their entire context.[43] The authors state: "we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts."[43]
U-Shaped Performance Curve: Model accuracy varies dramatically based on where relevant information appears:
Highest performance: Information at the very beginning or end of context
Lowest performance: Information in the middle of context
Performance drop: more than 30% accuracy decrease when relevant information shifts from start/end to middle positions[43]
This attention bias means that even if a document fits within the window, there is no guarantee the model will effectively use information in the middle sections, making it less reliable for tasks where critical details may appear anywhere in a long document. The finding replicated across six model families (GPT-3.5-Turbo, GPT-4, Claude 1.3, LongChat-13B, MPT-30B, and Cohere Command), and even GPT-4, the strongest model tested, exhibited the same U-shaped curve.[43]
Multi-Document QA Experiments: When presented with multiple documents where only one contains the answer:
GPT-3.5-Turbo-16K: Severe degradation for middle documents
Claude-1.3 (100K): Nearly perfect performance across all positions (notable exception)
Critically, simply extending context windows doesn't solve this problem. Models with 100K+ token windows exhibit the same U-shaped performance pattern; they can process more tokens but don't effectively use middle content. This suggests the issue is architectural rather than merely a capacity limitation.
Memory Bandwidth Bottleneck: Modern inference is primarily limited by memory bandwidth rather than compute. Loading the KV cache from HBM to SRAM for each generated token becomes the dominant cost, especially for long contexts.
Quadratic Cost Scaling: While techniques like FlashAttention reduce memory overhead, the fundamental O(n²) time complexity remains for full attention. Processing 128K tokens requires significantly more attention computations than processing shorter sequences.
Production Costs: Major cloud providers price long-context processing with significant premiums:
Anthropic: Higher pricing tiers for contexts exceeding 200K tokens
Google: Context caching features to amortize costs for repeated long prompts[44]
Time-to-First-Token (TTFT): The prefill phase (processing the initial prompt) grows with context length. Processing very long contexts can take several seconds before generation begins, impacting user experience for interactive applications.
Redundant Information: Research shows that as context grows beyond certain thresholds, redundant information increasingly hampers performance.[45] The model's limited "attention budget" gets diluted across many tokens, reducing focus on the most relevant information.
Attention Dilution: With fixed model capacity distributed across thousands or millions of tokens, the effective "attention budget" per token decreases as context length increases.
Training-Inference Mismatch: Most models train on shorter sequences (4K-32K tokens) and extend to longer contexts through positional encoding interpolation techniques. This mismatch can lead to degraded performance at extreme context lengths.
Legal Document Review: Models with 200K+ token context windows can process entire legal contracts, court filings, or patent applications in a single pass. Law firms use Claude Enterprise (with extended context) to analyze multi-document case files, identifying key clauses and potential issues without manual chunking.
Academic Research: Researchers upload multiple papers simultaneously, asking models to synthesize findings across literature. Models with million-token contexts can process dozens of academic papers at once, identifying trends, contradictions, and research gaps.
Medical Records: Healthcare providers analyze patient histories spanning years, combining clinical notes, lab results, and imaging reports. Long context windows enable comprehensive analysis while maintaining HIPAA compliance through proper access controls.
Codebase Understanding: Models like GPT-4.1 and Claude Sonnet 4 with large context windows can ingest entire codebases (tens of thousands of lines of code), enabling:
Cross-file refactoring with full dependency awareness
Bug identification requiring understanding of interactions across modules
Documentation generation that captures system-wide architecture
Code review with complete project context
SWE-bench Performance: Modern models demonstrate real-world software engineering capability on benchmarks like SWE-bench Verified, with top models achieving 70-80% accuracy on verified tasks that require understanding and modifying real GitHub repositories.
Book Writing: Authors use 128K+ context windows to maintain consistency across novel chapters, tracking character development, plot threads, and world-building details throughout lengthy works.
Screenplay Development: Hollywood writers leverage long context for multi-episode story arcs with large ensemble casts, ensuring character consistency and plot continuity across entire seasons.
Multi-Turn Conversations: Customer service bots maintain context across extended conversations. A 32K-token window enables approximately 50-100 conversation turns with full history, preventing repetitive questions and maintaining conversational coherence.
Technical Support: Support agents assisted by AI can reference entire conversation histories, previous tickets, documentation, and knowledge base articles simultaneously, providing more informed and contextually appropriate assistance.
Video Analysis: Models like Gemini 2.0 Pro process extended video content for applications including:
Content moderation across long videos
Sports game analysis with play-by-play understanding
Educational video summarization
Security footage review
Audio Processing: Long-context models handle extended audio recordings, enabling:
Podcast transcription and summarization
Meeting minutes generation
Audio content search and indexing
Multi-speaker conversation analysis
| Model | Context Window | Output Limit | Released |
|---|---|---|---|
| GPT-5 | 400,000 tokens (272K in + 128K out) | 128,000 tokens | August 2025 |
| GPT-4.1 | ~1,000,000 tokens | ~32,000 tokens | April 2025 |
| GPT-4o | 128,000 tokens | 16,384 tokens | May 2024 |
| Claude Opus 4.6 / Sonnet 4.6 | 1,000,000 tokens | 64,000 tokens | 2026 |
| Claude Sonnet 4.5 | 200,000 tokens | 64,000 tokens | September 2025 |
| Claude Opus 4.1 | 200,000 tokens | 64,000 tokens | August 2025 |
| Gemini 3.1 Pro | 1,048,576 tokens | 65,536 tokens | February 2026 |
| Gemini 2.5 Pro | ~1,000,000 tokens | ~65,000 tokens | 2025 |
| Gemini 2.0 Pro | ~2,000,000 tokens | ~8,000 tokens | 2025 |
| Llama 4 Scout | 10,000,000 tokens | N/A | April 2025 |
| Llama 4 Maverick | 1,000,000 tokens | N/A | April 2025 |
| MiniMax-Text-01 | 4,000,000 tokens | N/A | January 2025 |
| Mistral Large 2 | 128,000 tokens | N/A | July 2024 |
The GPT-5 series (released August 7, 2025) is OpenAI's flagship offering. The API version accepts up to 272,000 input tokens and returns up to 128,000 output tokens, for a total documented context window of 400,000 tokens:[29][30]
GPT-5: Standard model with unified reasoning
GPT-5-mini: Cost-efficient variant maintaining strong performance
GPT-5-nano: Smallest and fastest model in the family
The GPT-4.1 series (April 2025) features approximately 1 million tokens of context across all variants (GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano).[28] These models show particular strength in coding tasks and long-context comprehension.
The GPT-4o series continues with 128,000-token context windows, including GPT-4o-mini as a cost-efficient option.[46]
Claude Opus 4.6 and Claude Sonnet 4.6 (2026): 1,000,000 tokens as the generally available standard context window, with 64,000-token output maximum. The 1M context window became GA across Claude API, Amazon Bedrock, and Vertex AI, replacing the earlier beta-only access.[56]
Claude Sonnet 4.5 (September 2025): 200,000 tokens standard with 64,000-token output maximum, and a 1,000,000-token window available via the beta header context-1m-2025-08-07. Achieves 77.2% on SWE-bench Verified.[47]
Claude Opus 4.1 (August 2025): 200,000 tokens, optimized for complex reasoning tasks. Achieves 74.5% on SWE-bench Verified.
Claude Haiku 4.5: 200,000 tokens, fastest model with competitive pricing for high-throughput applications.
Claude Enterprise: Previously offered 500,000 tokens with GitHub integration and enterprise security features.[48]
Gemini 3.1 Pro (February 19, 2026): Features a 1,048,576-token context window with up to 65,536 output tokens. The model supports processing up to 900 images, 8.4 hours of audio, or 1 hour of video per prompt, and achieved 77.1% on ARC-AGI-2, targeting agentic workflows and large document analysis.[55]
Gemini 2.5 Pro: Features approximately 1 million tokens context window with extended thinking capabilities and multimodal processing.[49]
Gemini 2.0 Pro (Experimental): Supports up to 2 million tokens, among Google's largest context windows for experimental use cases.[49]
Gemini 2.0 Flash: 1,000,000 tokens with multimodal input support and optimized for lower latency applications.
Llama 4 Scout (April 5, 2025): 10,000,000 tokens, unprecedented for open-source models. With 109B parameters (17B active via MoE), it demonstrates that ultra-long context is achievable in open models.[3]
Llama 4 Maverick (April 5, 2025): 1,000,000 tokens with 400B parameters, competitive with closed-source models on many benchmarks.
Llama 3.2: 128,000 tokens for the 1B and 3B parameter variants, bringing long-context capability to smaller models suitable for edge deployment.
MiniMax-Text-01 (January 2025): 4,000,000 tokens with 456B parameters, the longest context from a Chinese AI startup at the time, demonstrating global competition in long-context capabilities.[32]
Magic.dev LTM-2-Mini (August 2024): Claims 100,000,000 tokens context window, described by Magic as equivalent to "10 million lines of code, or 750 novels," though practical performance at such scales requires verification.[33]
Mistral Large 2 (July 2024): 128,000 tokens with 123B parameters, competitive in multilingual tasks and code generation.[26]
The advent of multi-million token context windows has sparked debate about whether Retrieval-Augmented Generation (RAG) will become obsolete or remain complementary to long context. In practice, RAG and long context are converging rather than competing.
RAG decouples the knowledge source from the context window through a two-stage process:[50]
Retrieval: When a user submits a query, the system uses an information retrieval component (often vector search) to find the most relevant snippets from a large external knowledge base
Generation: These retrieved snippets are added to the prompt and fed into the LLM's context window, which then generates a response based on this focused information
Despite massive context windows, RAG maintains several key advantages:[51]
Scalability and Cost: Enterprise knowledge bases can contain petabytes of data, far exceeding even the most ambitious context windows. RAG scales to virtually unlimited data sizes and is more cost-effective because the LLM only processes small, relevant chunks of text per query.
Data Freshness and Dynamism: RAG systems can connect to real-time databases and information sources, ensuring responses are based on the most up-to-date information and overcoming the "knowledge cutoff" problem inherent in a model's static training data.
Performance and Reliability: By pre-filtering information and providing only the most relevant context, RAG improves the signal-to-noise ratio, helping to mitigate the "lost in the middle" problem and leading to more accurate responses.
Security and Access Control: In enterprise settings, RAG allows for robust security. Access controls can be implemented at the retrieval step, ensuring the LLM only sees data the specific user is authorized to view.
Rather than being competitors, RAG and long context windows are increasingly seen as complementary technologies that converge into a powerful hybrid architecture. In this model, RAG acts as a coarse-grained filter, retrieving a large but relevant set of documents from a massive corpus. This curated set can then be fed into a million-token model for deep, cross-document reasoning and synthesis, combining RAG's scalability with long context's analytical depth.
Hybrid Architectures: Combining transformers with alternative mechanisms:
State Space Models (SSMs): Models like Mamba achieve linear complexity while maintaining strong performance
Selective Attention: Dynamic determination of which tokens warrant full attention
Hierarchical Processing: Multi-level architectures processing information at different granularities[52]
Retrieval-Augmented Long Context: Future systems will likely combine retrieval with long context windows, using retrieval to identify relevant passages then processing within long contexts for deep synthesis.
Specialized Accelerators: Future AI chips will incorporate:
Dedicated attention computation units optimized for sparse patterns
Enhanced on-chip memory for larger SRAM caches
Custom interconnects for efficient KV cache distribution
Memory Innovations: Advances in HBM3, HBM4, and 3D-stacked memory will enable higher bandwidth and larger capacity, reducing the memory bottleneck that currently limits context window scaling.
Learned Sparsity: Rather than hand-crafted sparse attention patterns, models will learn which tokens require attention through neural architecture search and training-time optimization.
Token Pruning: Intelligent removal of redundant tokens through adaptive compression, maintaining semantic content while reducing computational load.[53]
Quantization Advances: Beyond FP8 and INT8 to 4-bit and 2-bit KV cache quantization with adaptive precision based on token importance.
Recurrent Memory Mechanisms: Approaches like Infini-attention point toward models that maintain bounded memory while processing unlimited sequences through selective compression and retention.[54]
Streaming Inference: Processing continuous streams without fixed context boundaries, with selective forgetting mechanisms that prioritize important information while discarding redundant details.
Addressing Lost in Middle: Active research focuses on:
Architectural modifications ensuring uniform attention across positions
Training procedures emphasizing middle-context utilization
Hybrid attention patterns that maintain global awareness[43]
Evaluation Methodologies: Development of better benchmarks beyond synthetic needle-in-haystack tests, including:
Multi-hop inference requiring integration of distant information
Domain-specific evaluations in legal, medical, and scientific contexts
Real-world task performance metrics
Transformer (machine learning model)
Attention mechanism
Claude (language model)
Gemini (language model)
Llama (language model)
Token (natural language processing)