# YaRN

> Source: https://aiwiki.ai/wiki/yarn
> Updated: 2026-10-27
> Fact-checked: 2026-11-04
> Categories: AI Inference, Deep Learning, Model Architecture
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "YaRN." aiwiki.ai, 27 Oct 2026. https://aiwiki.ai/wiki/yarn
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**YaRN** (Yet another RoPE extensioN method) is a compute-efficient method for extending the [context window](https://aiwiki.ai/wiki/context_window) of [large language model](https://aiwiki.ai/wiki/large_language_model)s that use [Rotary Position Embeddings](https://aiwiki.ai/wiki/rotary_position_embedding) (RoPE). Introduced by Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole in a paper submitted to [arXiv](https://aiwiki.ai/wiki/arxiv) on August 31, 2023 (arXiv:2309.00071), YaRN combines selective frequency interpolation with an attention temperature scaling mechanism to enable models to process sequences far longer than their original pre-training context window while requiring only a small amount of additional fine-tuning.[1] The paper reports that YaRN requires 10 times fewer tokens and 2.5 times fewer training steps than previous context-extension methods, and it was accepted at the [International Conference on Learning Representations](https://aiwiki.ai/wiki/iclr) (ICLR 2024).[1] The approach was subsequently adopted for context extension in several leading open-weight models, including [DeepSeek V3](https://aiwiki.ai/wiki/deepseek_v3), [Qwen 3](https://aiwiki.ai/wiki/qwen_3) (as an optional setting) and [Mistral Large](https://aiwiki.ai/wiki/mistral_large) 3, while [Llama 3.1](https://aiwiki.ai/wiki/llama_3_1) uses a closely related frequency-scaling scheme of its own.[6][13][14][15]

The paper states the method's core advantage directly: YaRN is "a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods," and shows that with it "LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow."[1]

The name is a self-aware reference to the proliferation of RoPE scaling techniques in 2023; at the time of publication, the field had already seen [Position Interpolation](https://aiwiki.ai/wiki/position_interpolation), NTK-aware scaling, and several community variants.[1]

## Background

### How do transformers encode token position?

Transformer models have no inherent notion of token order. Position information must be injected explicitly, either as learned embeddings added to token representations, or through mechanisms that modify the attention computation itself. Early large language models such as [GPT-2](https://aiwiki.ai/wiki/gpt-2) used learned absolute positional embeddings, which do not generalize beyond the maximum training length. Sinusoidal positional encodings (Vaswani et al., 2017) were proposed partly in the hope that they would extrapolate, but in practice also degrade beyond training length.[4][10] [ALiBi](https://aiwiki.ai/wiki/alibi) (Attention with Linear Biases, Press et al., 2022), used in models such as BLOOM, adds a fixed (non-learned), distance-proportional penalty with head-specific slopes to attention [logits](https://aiwiki.ai/wiki/logits) and extrapolates more gracefully, but it is not compatible with the RoPE architecture.[4]

### Rotary Position Embeddings

Rotary Position Embeddings (RoPE), introduced by Su et al. in the RoFormer paper (arXiv:2104.09864), became the dominant [positional encoding](https://aiwiki.ai/wiki/positional_encoding) scheme in open-weight [foundation models](https://aiwiki.ai/wiki/foundation_models), including the entire LLaMA family, Mistral, [Qwen](https://aiwiki.ai/wiki/qwen), [DeepSeek](https://aiwiki.ai/wiki/deepseek), and others.[3] RoPE applies a two-dimensional rotation to consecutive pairs of dimensions in the query and key vectors before the attention dot product is computed. Each pair of dimensions $$d$$ is rotated by angle $$m \theta_d$$, where $$m$$ is the token position and $$\theta_d = \mathrm{base}^{-2d/D}$$. Here $$D$$ is the embedding dimension and base is typically 10,000. The inner product between a rotated query at position $$m$$ and a rotated key at position $$n$$ depends only on the relative displacement $$(m - n)$$, naturally encoding relative position.[3]

Because the rotation angle depends on both the position $$m$$ and the frequency $$\theta_d$$, RoPE creates a multi-scale positional representation. Lower-indexed dimension pairs have large $$\theta_d$$ values and therefore rotate rapidly as positions increase (high frequency, encoding local short-range structure). Higher-indexed dimension pairs have small $$\theta_d$$ values and rotate slowly (low frequency, encoding global long-range structure). This hierarchical frequency structure is central to why selective scaling across frequencies, as in YaRN, works better than uniform scaling.[1]

### Why do RoPE models fail beyond their training length?

Despite their advantages, RoPE-based models generalize poorly beyond their pre-training context length $$L_{\mathrm{train}}$$. When a model encounters token positions $$m$$ greater than $$L_{\mathrm{train}}$$, the rotary angles $$m \theta_d$$ for those positions lie entirely outside the range seen during training. The attention mechanism relies on inner products between rotated queries and keys; at unseen angles, the inner products become unpredictable and the model's ability to relate distant tokens breaks down. In practice this manifests as a sharp perplexity spike and loss of coherent output when sequence length exceeds the training maximum by more than a modest amount.[1] The YaRN paper frames this bluntly: such models "fail to generalize past the sequence length they were trained on."[1]

This limitation is significant for production use cases. A model trained with a 4K or 8K context window cannot reliably summarize long documents, process entire codebases, maintain coherence across book-length inputs, or handle many-turn extended conversations without expensive re-pre-training from scratch on longer sequences. As instruction-tuned models became widely deployed in 2023, the gap between typical training context lengths (4K to 8K tokens) and user demand for [long context](https://aiwiki.ai/wiki/long_context) (10K to 128K or more) became a pressing practical problem.

### Position Interpolation

The first widely adopted solution was Position Interpolation (PI), introduced by Chen et al. (arXiv:2306.15595) at Meta in June 2023.[2] The core idea is to scale down position indices at inference time: rather than feeding position $$m$$ directly into RoPE, use $$m L_{\mathrm{train}} / L_{\mathrm{target}}$$. The longest position in the extended context thus maps to the same rotary angle as the longest position seen during training, ensuring all angles remain within the trained distribution. With roughly 1,000 fine-tuning steps, PI allowed LLaMA models to extend to 32,768 tokens.[2]

However, PI applies uniform compression to all position indices and all frequency dimensions. At a scaling factor of $$s$$ (target context = $$s$$ times original), adjacent tokens are $$1/s$$ apart in the compressed position space. High-frequency RoPE dimensions, which rely on the difference between nearby positions to encode fine-grained local relationships, are particularly harmed: at 8x extension, the distance between adjacent tokens in position space becomes $$1/8$$ of what the model was trained on, effectively making it much harder to distinguish nearby tokens in those dimensions. This degradation grows with the extension factor. Perplexity at standard context lengths measurably increases in PI-extended models, reflecting the loss of local positional precision.[1]

### NTK-Aware Scaling

NTK-aware scaling (also called NTK-RoPE), which circulated as a community contribution in mid-2023, takes a different approach: instead of compressing position indices, it increases the RoPE base frequency. By setting $$\mathrm{base}_{\mathrm{new}} = \mathrm{base} \cdot \alpha^{D/(D-2)}$$ for a chosen scale factor $$\alpha$$, the frequencies of all dimensions are slowed down uniformly, so that positions up to $$L_{\mathrm{target}} \cdot \mathrm{base}_{\mathrm{ratio}}$$ still correspond to angles within a reasonable range. The name comes from [Neural Tangent Kernel](https://aiwiki.ai/wiki/neural_tangent_kernel) theory, which shows that networks have trouble learning high-frequency information from low-dimensional inputs: uniform PI-style scaling removes RoPE's high-frequency components, whereas the base change scales high frequencies less and low frequencies more.[1][8]

NTK-aware scaling works as a zero-shot approach, requiring no fine-tuning, because it avoids placing token positions at entirely unseen angles. When combined with fine-tuning, however, the YaRN paper reports that it yields inferior results to PI, because some dimensions are slightly extrapolated to out-of-bound values.[1] The limitation is that scaling the base uniformly still treats all frequency dimensions identically: high-frequency dimensions are slowed when their original frequencies were already well-suited for the training context length, and low-frequency dimensions may not be slowed enough at very large extension ratios.[1]

### The Motivation for YaRN

Both PI and NTK-aware scaling apply a single scalar to all frequency dimensions despite the fact that different dimensions serve fundamentally different roles. The YaRN authors identified this as the central failure mode: high-frequency dimensions should not be interpolated (they already generalize locally), while low-frequency dimensions must be interpolated to bring extended positions into the trained angular range.[1] An approach that could apply extrapolation to high frequencies and interpolation to low frequencies, with a smooth transition between the two regimes, would outperform either uniform strategy. YaRN provides exactly this, combined with a principled correction for the change in attention score distribution caused by any form of RoPE modification.[1]

## The YaRN Paper

### Who wrote the YaRN paper?

The four authors represent a collaboration between [Nous Research](https://aiwiki.ai/wiki/nous_research) and [EleutherAI](https://aiwiki.ai/wiki/eleutherai). Bowen Peng, the lead author, is listed with Nous Research.[1] Jeffrey Quesnelle is a co-founder of Nous Research, an [open-source AI](https://aiwiki.ai/wiki/open_source_ai) research organization focused on agent reliability, context-length extension, and novel training techniques; as of October 2026 he is its chief technology officer, and Dillon Rolnick is its chief executive.[11][12] Honglu Fan is listed with EleutherAI, the open research collective known for the GPT-NeoX and Pythia model series, and the University of Geneva.[1] Enrico Shippole is listed without an institutional affiliation.[1] The collaboration brought together practical open-source model development expertise (Nous Research's fine-tuning work on the LLaMA family) with fundamental research capabilities (EleutherAI's infrastructure and evaluation tooling).

### When was YaRN published?

The initial version of the paper was submitted on August 31, 2023.[1] A revised version (v2) was posted on November 1, 2023, and the paper was accepted at ICLR 2024.[1][9] A third arXiv version was posted on February 6, 2026.[1] The [GitHub](https://aiwiki.ai/wiki/github) repository (github.com/jquesnelle/yarn), maintained by Quesnelle, was created in June 2023, ahead of the paper, and had about 1,800 stars and 136 forks as of November 2026.[9] The repository includes training scripts, [DeepSpeed](https://aiwiki.ai/wiki/deepspeed) configuration files, evaluation scripts built on lm-evaluation-harness, and links to trained model weights published on [Hugging Face](https://aiwiki.ai/wiki/hugging_face) under appropriate licenses.[9]

### Core Claims

The paper's primary claims are:

1. YaRN requires roughly 10 times fewer fine-tuning tokens and 2.5 times fewer training steps than previous methods (primarily PI) to achieve equivalent or superior perplexity at extended context lengths.[1]
2. YaRN outperforms Position Interpolation, NTK-aware scaling, and NTK-by-parts (without temperature) after fine-tuning.[1]
3. A dynamic, zero-shot variant (Dynamic YaRN) achieves more than 2x context extension with no fine-tuning.[1]
4. YaRN models can extrapolate to contexts longer than their fine-tuning context length (train short, test long).[1]
5. Standard NLP benchmark performance (ARC, [HellaSwag](https://aiwiki.ai/wiki/hellaswag), MMLU, [TruthfulQA](https://aiwiki.ai/wiki/truthfulqa)) is minimally degraded by context extension.[1]

## How does YaRN work?

### NTK-by-Parts Interpolation

YaRN's first technical contribution is NTK-by-parts interpolation, a piecewise blending strategy that applies different scaling to different frequency dimensions based on each dimension's characteristic wavelength relative to the target context length.[1]

The wavelength of dimension pair $$d$$ is the sequence length at which that dimension completes one full $$2\pi$$ rotation cycle: $$\lambda_d = 2\pi / \theta_d$$. Dimensions with small index (high $$\theta_d$$) have short wavelengths and complete many full rotations within any non-trivial sequence. Dimensions with large index (low $$\theta_d$$) have long wavelengths, potentially much longer than the training context.[1]

The ramp function $$\gamma(r)$$ maps a wavelength ratio $$r$$ to a blending coefficient between 0 and 1:

- $$\gamma(r) = 0$$ when $$r$$ is less than $$\alpha$$ (low-frequency regime: the dimension makes fewer than alpha full rotations within the original context, so it is fully interpolated)
- $$\gamma(r) = 1$$ when $$r$$ is greater than $$\beta$$ (high-frequency regime: the dimension makes more than beta full rotations within the original context, so it is left unchanged)
- $$\gamma(r) = (r - \alpha) / (\beta - \alpha)$$ when $$\alpha$$ is less than or equal to $$r$$ is less than or equal to $$\beta$$ (transition zone)

Here $$r = L / \lambda_d$$ is the ratio of the original pre-training context length $$L$$ to the dimension's wavelength, that is, the number of full rotations the dimension makes within the original context. The blended modified frequency for dimension d is then:

$$
h(\theta_d) = (1 - \gamma(r)) (\theta_d / s) + \gamma(r) \theta_d
$$

where $$s$$ is the extension factor (target length divided by original length). When $$\gamma = 0$$, the frequency is divided by $$s$$ (full interpolation, as in PI). When $$\gamma = 1$$, the original frequency $$\theta_d$$ is preserved unchanged (no interpolation). In the transition zone, a weighted blend is applied.[1]

For LLaMA-family models, the authors found through empirical experimentation that $$\alpha = 1$$ and $$\beta = 32$$ worked well.[1] This means dimensions whose wavelength is longer than the original context length receive full interpolation, dimensions whose wavelength is shorter than $$1/32$$ of the context length receive no interpolation, and a linear ramp connects the two regimes. The paper says the values should be tuned case by case for other model families.[1]

### Attention Temperature Scaling

Modifying RoPE frequencies changes the distribution of the dot products between rotated queries and keys. When frequencies are compressed (interpolated), the inner products between query-key pairs at similar positions become smaller in magnitude, leading to a flatter attention distribution that treats all tokens more equally. When frequencies are extrapolated, the opposite effect can occur. Either way, the attention [softmax](https://aiwiki.ai/wiki/softmax) distribution diverges from what the model was trained on, leading to suboptimal behavior.[1]

YaRN's second contribution addresses this directly with an attention temperature correction.[1] The standard scaled dot-product attention computes $$\mathrm{softmax}(q^\top k / \sqrt{D})$$. YaRN introduces a temperature parameter $$t$$ that further scales the denominator, effectively modifying the sharpness of the attention distribution. Dividing the logits by $$t > 1$$ would flatten the attention distribution; YaRN's recommended values give $$t < 1$$, which multiplies the logits by $$1/t > 1$$ and sharpens it.[1]

The key finding of the paper is an empirical formula for the optimal temperature as a function of the extension factor $$s$$:

$$
\sqrt{1/t} = 0.1 \ln(s) + 1
$$

The authors fitted this logarithmic relationship on LLaMA 7B, 13B, 33B and 65B models without fine-tuning and note that the same values also apply fairly well to [Llama 2](https://aiwiki.ai/wiki/llama_2) 7B, 13B and 70B.[1] For example, at $$s = 16$$ (a 4K model extended to 64K), $$\sqrt{1/t} = 0.1 \ln(16) + 1 \approx 1.277$$, giving $$t \approx 0.613$$. Queries and keys are each scaled by about 1.277, so the attention logits are multiplied by $$1/t \approx 1.63$$, sharpening the distribution to counter the increase in attention entropy that comes with the modified RoPE frequencies.[1]

An important implementation detail: the temperature scaling can be absorbed into the query and key normalization step, adding zero overhead at inference time. Rather than modifying the softmax computation, implementations scale the query and key vectors by $$\sqrt{1/t}$$ before computing the dot product, which is mathematically equivalent and compatible with optimized attention kernels including [Flash Attention](https://aiwiki.ai/wiki/flash_attention).[1]

In Hugging Face-format configs such as DeepSeek V3's, the framework computes the attention multiplier $$\sqrt{1/t}$$ from the scale factor with this same formula; the optional `mscale` and `mscale_all_dim` fields are coefficients on the $$\ln(s)$$ term (both 1.0 for DeepSeek V3), not the multiplier itself.[14]

### Dynamic YaRN

A third contribution of the paper is the dynamic extension variant, which requires no fine-tuning.[1] In Dynamic YaRN, the scale factor $$s$$ is not fixed during deployment but computed on the fly for each forward pass as $$s = \max(1, l / L_{\mathrm{train}})$$, where $$l$$ is the length of the current input and $$L_{\mathrm{train}}$$ is the training context length. If the current input fits within the original training length, $$s = 1$$ and no scaling is applied, preserving original model behavior. As the sequence grows longer, $$s$$ increases and the ramp-blended interpolation is applied progressively.[1]

Dynamic scaling achieves more than 2x context extension with zero fine-tuning.[1] It is less accurate at extreme lengths than the fine-tuned static variant, but it provides a practical drop-in upgrade for any deployed RoPE model without requiring weight changes. [llama.cpp](https://aiwiki.ai/wiki/llama_cpp), [vLLM](https://aiwiki.ai/wiki/vllm), [Hugging Face Transformers](https://aiwiki.ai/wiki/transformers_library) and other inference frameworks support YaRN as a configurable rope_scaling option, but in its static form: Qwen's model documentation notes that the notable open-source frameworks "implement static YaRN," with a scaling factor that stays constant regardless of input length.[13]

## How does YaRN compare to other context-extension methods?

The following table summarizes how YaRN compares to the primary competing approaches for extending context in RoPE-based [language model](https://aiwiki.ai/wiki/language_model)s:

| Method | Fine-tuning | Frequency treatment | Attention correction | Typical extension | Notes |
|---|---|---|---|---|---|
| Position Interpolation | Required | Uniform (position scaling) | None | Up to 8x well, degrades beyond | Degrades local precision |
| NTK-Aware Scaling | Optional | Uniform (base rescaling) | None | 2-4x zero-shot | Better than PI zero-shot |
| NTK-by-Parts | Optional | Non-uniform (piecewise) | None | 4-8x | Partial solution |
| YaRN (static) | Light (400-600 steps) | Non-uniform (ramp) | Yes (temperature) | 8-32x reliably | Best perplexity fine-tuned |
| Dynamic YaRN | None | Non-uniform (adaptive) | Yes | 2x+ reliably | Best zero-shot |
| [LongRoPE](https://aiwiki.ai/wiki/longrope) | Optional | Per-dimension search | None | 8x without fine-tuning; 2048K with fine-tuning | Automated, no heuristics |

A central empirical result from the paper is that YaRN significantly outperforms Position Interpolation on extended perplexity while also avoiding PI's degradation on short contexts. On the original LLaMA 7B extended to 32K ($$s = 16$$) with 400 fine-tuning steps, YaRN achieves a perplexity of 2.77 at 32K versus PI's 3.57.[1] NTK-aware scaling, despite being reasonable zero-shot, actually performs poorly after fine-tuning (perplexity 8.49), suggesting that its uniform scaling creates a poor initialization for fine-tuning. NTK-by-parts alone (without temperature scaling) achieves 2.81, just slightly above YaRN's 2.77, demonstrating that the temperature correction provides a meaningful additional improvement.[1]

PI fine-tuned models show measurably higher perplexity on standard-length inputs compared to the original model.[2] YaRN's frequency partitioning avoids this regression by preserving high-frequency dimensions unchanged, which are the most important for short-range token discrimination.[1]

### Comparison with ALiBi

ALiBi takes a fundamentally different architectural approach: it encodes position through a linear bias added to attention logits rather than modifying query and key representations.[4] ALiBi extrapolates gracefully to contexts somewhat longer than training and requires no position-specific fine-tuning.[4] However, it is architecturally incompatible with RoPE: models already trained with RoPE cannot adopt ALiBi without structural changes and re-training. YaRN specifically targets the RoPE ecosystem, where the majority of open-weight foundation models live.

## Empirical Results

### Perplexity on Extended Contexts

The paper evaluates perplexity using a sliding window approach with stride S = 256 on the Proof-pile and GovReport datasets (PG-19 was the fine-tuning data).[1] At 32K context length with 400 fine-tuning steps, YaRN-extended LLaMA 7B achieves perplexity of 2.77, substantially below both PI (3.57) and NTK-aware (8.49). At 128K context with scale factor $$s = 32$$, Llama 2 7B achieves perplexity of 2.37, compared to 2.71 for [Code Llama](https://aiwiki.ai/wiki/code_llama) (which uses an NTK-based extension).[1]

On the GovReport benchmark across 50 long government documents at a 32K sliding window, YaRN at $$s = 16$$ achieves perplexity of 3.59 and at $$s = 32$$ achieves 3.64, versus Code Llama's 4.44.[1]

### PassKey Retrieval

PassKey retrieval, a synthetic task, tests whether a model can locate a short numeric passkey planted at a random position within a long document of irrelevant filler text. It is a simple diagnostic for whether the model can attend to arbitrary positions in a long context rather than defaulting to a recency bias. YaRN Llama 2 7B and 13B extended to 128K context both achieved 99.4% accuracy on PassKey retrieval, indicating effective attention across the full context window without positional blind spots.[1]

### Standard Benchmark Retention

To verify that context extension does not degrade standard language understanding capabilities, the paper evaluates YaRN-extended Llama 2 7B on ARC-Challenge, HellaSwag, MMLU, and TruthfulQA. The results show minimal regression:[1]

| Benchmark | Baseline | YaRN s=16 | YaRN s=32 |
|---|---|---|---|
| ARC-Challenge | 53.1 | 52.3 | 52.1 |
| [HellaSwag](https://aiwiki.ai/wiki/hellaswag) | 77.8 | 78.8 | 78.4 |
| MMLU | 43.8 | 42.5 | 41.7 |
| [TruthfulQA](https://aiwiki.ai/wiki/truthfulqa) | 39.0 | 38.2 | 37.3 |

The small regressions on MMLU and TruthfulQA, and the slight improvement on HellaSwag, indicate that YaRN's temperature-corrected interpolation preserves the model's general knowledge and reasoning capabilities through context extension.[1]

### How much compute does YaRN save?

YaRN's fine-tuning requirements are modest by the standards of context extension research. The paper's Table 4 lists 256 A100 GPU-hours to extend Llama 2 7B to 64K (400 steps at $$s = 16$$) and 128 more for the additional 200 steps to reach $$s = 32$$ (128K), with 128 GPU-hours for the original LLaMA 7B at 32K.[1] For comparison it lists 640 GPU-hours for the PI model of Chen et al. (16K), 6,400 for Code Llama's NTK-aware extension (about 100K) and 64,000 for the NTK-aware model of Xiong et al. (about 50K).[1] The Llama models were trained on the PG19 books dataset tokenized into 64K-token (65,536) chunks; the Mistral models used a mix of Together's Long-Data-Collections.[1][9] Fine-tuning with YaRN on less than 0.1% of the original pre-training data volume achieves the reported perplexity numbers.[1]

### Models Released

The YaRN repository published fine-tuned model variants for:[9]
- LLaMA 2 7B at 64K and 128K context
- LLaMA 2 13B at 64K and 128K context
- LLaMA 2 70B at 32K context
- Mistral 7B at 64K and 128K context
- SOLAR 10.7B at 32K and 64K context

All models were published on Hugging Face under appropriate licenses and the repository linked to their checkpoints.[9]

## Which models use YaRN?

### Llama 3.1

Llama 3.1, released by Meta in July 2024, extended the context window from the 8K of [Llama 3](https://aiwiki.ai/wiki/llama_3) to 128K tokens.[7] Meta's approach used a multi-stage long-context pre-training process with six progressive stages from 8K to 128K across 800 billion training tokens.[7] The RoPE base frequency was dramatically increased to 500,000 (versus 10,000 in the original LLaMA) to provide a better initialized positional encoding for extended contexts.[7] The released Llama 3.1 checkpoints do not use YaRN's configuration: they specify a separate "llama3" RoPE scaling type (factor 8, low_freq_factor 1, high_freq_factor 4, original context 8,192) that, like YaRN's NTK-by-parts component, leaves high-frequency dimensions unchanged, divides low-frequency ones by the factor and blends in between, but applies no attention-temperature correction.[14] The Llama 3 technical report does not cite YaRN,[7] although some later papers, such as LongRoPE2, loosely describe Llama 3.1 as adopting YaRN.[16]

### DeepSeek V3

DeepSeek V3, the large mixture-of-experts model released by DeepSeek in December 2024 (arXiv:2412.19437), applies YaRN explicitly in two post-pre-training context extension phases.[6] After initial pre-training, the model undergoes two sequential 1,000-step fine-tuning phases: in the first, the sequence length target is 32K tokens with a [batch size](https://aiwiki.ai/wiki/batch_size) of 1,920; in the second, it is extended to 128K tokens with a batch size of 480.[6] Both phases use the YaRN temperature formula $$\sqrt{1/t} = 0.1 \ln(s) + 1$$, applied to the decoupled shared key within DeepSeek's [Multi-Head Latent Attention](https://aiwiki.ai/wiki/multi-head_latent_attention) (MLA) architecture.[6] The same YaRN configuration was inherited from DeepSeek V2, indicating that the technique's stability enabled direct reuse across architectural generations of the model series.[6]

### Qwen 2.5 and Qwen 3

Alibaba's Qwen family uses YaRN as an optional context extension mechanism. The native context length of most Qwen 3 models is 32,768 tokens. Applying YaRN rope scaling with factor 4.0 and original_max_position_embeddings = 32,768 extends the effective context to 131,072 tokens.[13] The standard config entry is `rope_scaling: {factor: 4.0, original_max_position_embeddings: 32768, type: "yarn"}`. Qwen 2.5 followed the same pattern. The design choice to make YaRN optional, off by default and activated via configuration, reflects a practical trade-off: standard-length inputs benefit from the full pre-trained perplexity without context compression artifacts, while users needing longer contexts can activate the extension.

### Mistral

The second (November 2023) version of the YaRN paper included extended Mistral 7B v0.1 variants at 64K and 128K context, demonstrating that the technique transferred across model families from LLaMA to Mistral.[1] The Mistral architecture uses [Grouped Query Attention](https://aiwiki.ai/wiki/grouped_query_attention) and (in v0.1) a [sliding window attention](https://aiwiki.ai/wiki/sliding_window_attention) pattern, which required some adaptation. Mistral 7B v0.2, released by [Mistral AI](https://aiwiki.ai/wiki/mistral_ai) in December 2023, adopted a 32K context window with a substantially increased rope_theta of 1,000,000 and removed sliding window attention entirely. Mistral AI has not said whether this change drew on the YaRN results. Its later Mistral Large 3 (December 2025) does use YaRN: the published parameters specify a YaRN block with factor 36, alpha 1, beta 32 and an original context of 8,192 tokens.[15]

### Broader Open-Source Ecosystem

Beyond the major commercial model families, YaRN found wide adoption in the independent fine-tuning community. Because the technique requires very little compute, individual researchers and smaller organizations could apply it to any RoPE-based model without the [GPU cluster](https://aiwiki.ai/wiki/gpu_cluster)s required for full pre-training. The YaRN repository's training scripts (built on DeepSpeed and Hugging Face accelerate), together with the publicly available tokenized training datasets, made long-context fine-tuning accessible at the scale of a single multi-GPU workstation.[9] Inference framework integration (llama.cpp, vLLM, Hugging Face Transformers) followed shortly after, further democratizing deployment.

## LongRoPE and Successors

### LongRoPE

[LongRoPE](https://aiwiki.ai/wiki/longrope), introduced by Ding et al. at [Microsoft Research](https://aiwiki.ai/wiki/microsoft_research) (arXiv:2402.13753, accepted at ICML 2024), extended the YaRN insight while addressing its primary limitation: the reliance on human-designed heuristics (alpha and beta) for frequency partitioning.[5] LongRoPE replaces the ramp function with an evolutionary search algorithm that discovers per-dimension rescaling factors $$\lambda_i$$ automatically, optimized to minimize perplexity on the target context length.[5] It also introduces a second type of non-uniformity: initial token positions receive less aggressive interpolation than later positions, preserving the strong attention to sequence beginnings that is important for instruction following and context framing.[5]

LongRoPE was integrated into [Microsoft](https://aiwiki.ai/wiki/microsoft)'s [Phi-3](https://aiwiki.ai/wiki/phi_3) model family and demonstrated context extension to 2 million tokens.[5] The LongRoPE paper reports that its searched factors significantly outperform PI and YaRN both before and after fine-tuning LLaMA2-7B, and that without fine-tuning YaRN's perplexity spikes after 7K tokens in an 8K target window.[5] The progression from PI to NTK to NTK-by-parts to YaRN to LongRoPE illustrates the field's movement from uniform scaling to hand-crafted non-uniform rules to automated per-dimension optimization.

### LongRoPE2

LongRoPE2 (arXiv:2502.20082) uses an evolutionary search guided by "needle-driven" perplexity plus mixed context window training, and reports extending LLaMA3-8B to a 128K effective context while retaining over 98.5% of short-context performance with 10B training tokens.[16] It argues that the theoretical scaling bound YaRN adheres to in higher RoPE dimensions is often sub-optimal, reporting that NTK and LongRoPE outperform YaRN on both short- and long-context tasks when extending Phi3-mini to 128K.[16]

### Dynamic Scaling in Inference Frameworks

The dynamic scaling concept from the YaRN paper became a standard configuration option in production inference frameworks. llama.cpp, vLLM, and Hugging Face Transformers all support a `"yarn"` rope_scaling type, which they implement as static YaRN; their `"dynamic"` option is Dynamic NTK scaling rather than Dynamic YaRN.[13][14] Configuration files for widely used models, including DeepSeek V3 and Mistral Large 3, specify YaRN parameters (Phi-3's long-context models use a separate `longrope` type) that the inference backend reads to apply the appropriate frequency blending and temperature correction at runtime without modifying model weights.

## What are the limitations of YaRN?

**Heuristic parameter selection.** The ramp function parameters alpha and beta must be chosen empirically for each model family. Although $$\alpha = 1$$ and $$\beta = 32$$ work well for LLaMA-family models, they are not theoretically derived and may be sub-optimal for models with significantly different architectures, base frequencies, or pre-training distributions.[1] LongRoPE's automated search was directly motivated by this limitation.[5]

**Perplexity degradation on standard benchmarks.** Even with minimal regression, extending context introduces a small but measurable perplexity increase on short-context benchmarks at higher extension factors. At $$s = 32$$, MMLU drops from 43.8 to 41.7 and TruthfulQA from 39.0 to 37.3.[1] Users who do not need long context should use model variants without YaRN applied, as the temperature and frequency modifications have a net effect on standard-length inference even if small.

**Extrapolation ceiling.** YaRN enables fine-tuning at one context length and inference at a somewhat longer length, but this extrapolation is not unlimited. Very large extension ratios (32x or more) require fine-tuning at or near the target length. In the paper, the $$s = 32$$ model fine-tuned on 64K-token sequences extrapolated to 128K (twice its training length), while the $$s = 16$$ models' perplexity rose above 10 beyond their 64K target.[1]

**RoPE specificity.** YaRN applies only to RoPE-based models. Models using learned absolute positional embeddings, sinusoidal encodings, ALiBi, or other positional schemes require entirely different approaches. While the RoPE ecosystem covers most major open-weight models as of 2024, this limits YaRN's applicability to that subset.

**Inference memory cost unchanged.** YaRN dramatically reduces the training cost of context extension but has no effect on inference memory cost. Processing a 128K token context still requires the full key-value cache for all 128K positions. In standard attention, [KV cache](https://aiwiki.ai/wiki/kv_cache) memory scales linearly with sequence length (quadratic attention computation is addressed by Flash Attention, but memory remains linear per layer). Practical deployment of YaRN-extended models at very long contexts still requires careful KV cache management, quantized caches, or architectural techniques such as sliding window attention or grouped key-value sharing.

**Static versus dynamic trade-off.** The static (fine-tuned) and dynamic variants serve different deployment needs. Static YaRN gives the best quality but requires a separate trained model variant for each target context length. Dynamic YaRN avoids fine-tuning but shows higher perplexity at very long contexts.[1] Qwen's documentation notes that the notable open-source inference frameworks implement only static YaRN, so operators must pre-specify the scaling factor at model load time rather than adapting it per request.[13]

**Temperature formula generalization.** The empirical formula $$\sqrt{1/t} = 0.1 \ln(s) + 1$$ was fitted on LLaMA 7B to 65B and reported to apply fairly well to Llama 2 7B to 70B.[1] For model families with substantially different pre-training data, [tokenization](https://aiwiki.ai/wiki/tokenization), or architecture, the optimal temperature may differ. DeepSeek V2 and V3 used the same formula, suggesting it generalizes reasonably, but the theoretical basis for this generalization is not fully established.[6]

## See Also

- [Rotary Position Embedding](https://aiwiki.ai/wiki/rotary_position_embedding)
- [Context Window](https://aiwiki.ai/wiki/context_window)
- [Long Context](https://aiwiki.ai/wiki/long_context)
- [Flash Attention](https://aiwiki.ai/wiki/flash_attention)
- [Grouped Query Attention](https://aiwiki.ai/wiki/grouped_query_attention)
- [Llama 3.1](https://aiwiki.ai/wiki/llama_3_1)
- [Qwen 3](https://aiwiki.ai/wiki/qwen_3)
- [Mistral Large](https://aiwiki.ai/wiki/mistral_large)
- [DeepSeek V3](https://aiwiki.ai/wiki/deepseek_v3)

## References

1. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. (2023). "YaRN: Efficient Context Window Extension of Large Language Models." arXiv:2309.00071. ICLR 2024. https://arxiv.org/abs/2309.00071
2. Chen, S., Wong, S., Chen, L., and Tian, Y. (2023). "Extending Context Window of Large Language Models via Positional Interpolation." arXiv:2306.15595. ICLR 2024.
3. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv:2104.09864.
4. Press, O., Smith, N. A., and Lewis, M. (2022). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." ICLR 2022.
5. Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. (2024). "LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens." arXiv:2402.13753. ICML 2024.
6. DeepSeek-AI (2024). "DeepSeek-V3 Technical Report." arXiv:2412.19437.
7. Meta AI (2024). "The Llama 3 Herd of Models." arXiv:2407.21783.
8. EleutherAI Blog. "Extending the RoPE." https://blog.eleuther.ai/yarn/
9. jquesnelle/yarn GitHub repository. https://github.com/jquesnelle/yarn
10. Vaswani, A. et al. (2017). "Attention Is All You Need." NeurIPS 2017.
11. Quesnelle, J. "Homepage of Jeffrey Quesnelle." jeffq.com, accessed October 27, 2026. https://jeffq.com/
12. Rolnick, D. (2026). "A Note on Our Fundraise." Nous Research, October 2026. https://nousresearch.com/a-note-on-our-fundraise
13. Qwen Team. "Qwen3-8B" model card, Hugging Face. https://huggingface.co/Qwen/Qwen3-8B
14. Hugging Face. "modeling_rope_utils.py" (RoPE scaling implementations: yarn, dynamic, llama3), Transformers library source. https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_rope_utils.py
15. Mistral AI. "Mistral-Large-3-675B-Instruct-2512" params.json, Hugging Face. https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512/blob/main/params.json
16. Shang, N., Zhang, L. L., Wang, S., Zhang, G., Lopez, G., Yang, F., Chen, W., and Yang, M. (2025). "LongRoPE2: Near-Lossless LLM Context Window Scaling." arXiv:2502.20082. https://arxiv.org/abs/2502.20082