Rotary Position Embedding

Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces, so that the query-key dot product depends on the two positions only through their difference. It was introduced by Jianlin Su and colleagues at Zhuiyi Technology in the paper "RoFormer: Enhanced Transformer with Rotary Position Embedding", submitted to arXiv on 20 April 2021 and published in Neurocomputing (volume 568, article 127063) in February 2024.^[1] In the authors' words, "the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation."^[1] The rotation angles are fixed functions of position, so RoPE adds no learnable parameters.^[1]

The RoFormer abstract names three properties that the rotation scheme provides: "the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding."^[1] Su first sketched the idea, which he then called a "fusion" style of encoding, in a Chinese-language blog post on his site Kexue.fm on 3 February 2021, and described it in full as rotary position embedding in a post of 23 March 2021 that also announced the open-source RoFormer model.^[2] On 20 April 2021 EleutherAI published an English-language explainer that described RoPE as "a new type of position encoding that unifies absolute and relative approaches", reported that across its experiments it "either matches or surpasses all other methods currently available for injecting positional information into transformers", and said the group had implemented it in x-transformers, GPT-Neo, GPT-NeoX and Mesh Transformer JAX.^[19] GPT-J-6B, released from the Mesh Transformer JAX codebase in June 2021, applied RoPE to 64 of each attention head's 256 dimensions.^[24]

Google's PaLM (2022) used RoPE,^[15] as did Meta's LLaMA (February 2023) and Llama 2 (July 2023).^[9][10] As of October 2026 RoPE, or a modified form of it, is the positional scheme in most major open-weight large language model families, including Llama, Mistral, Qwen, Gemma, DeepSeek and OpenAI's gpt-oss (see the adoption table below).^{[30][31][32][36][41][43]} Several recent designs modify it: DeepSeek's Multi-head Latent Attention applies RoPE only to a small decoupled part of each query and key,^[17] Llama 4, Cohere's Command A and Hugging Face's SmolLM3 drop it from every fourth layer,^[37][40][42] and multimodal models such as Qwen2-VL split the rotation across time, height and width.^[34]

Background and motivation

Before RoPE, transformer models typically used either sinusoidal positional encodings (Vaswani et al., 2017) or learned positional embeddings (BERT, GPT-2).^[3] Both are absolute positional methods: they assign a fixed vector to each position in the sequence, which is added to the token embedding. While these approaches work, they have notable limitations.

Absolute positional encodings do not directly encode the distance between tokens. If a model needs to learn that "a verb typically follows its subject by one or two positions," it must extract this relative information indirectly from absolute position vectors. Learned embeddings also impose a hard maximum sequence length, since there is no embedding for positions beyond the training maximum.

Relative positional methods make attention depend on the distance between query and key positions: Shaw et al. (2018) add learned relative-position embeddings inside the attention computation, Transformer-XL (Dai et al., 2019) uses sinusoidal relative terms with learned global biases, and T5 (Raffel et al., 2020) adds a learned scalar bias for each bucket of relative distance.^[4][12][13] ALiBi (Press et al., 2022) adds a fixed, head-specific penalty to attention scores that grows linearly with distance.^[5] However, these methods typically require additional parameters or modify the attention computation in ways that can complicate efficient implementation.

Su et al. sought a method that would encode absolute position (so each token knows where it is), naturally produce relative position information in the attention scores (so the model knows how far apart tokens are), require no additional parameters, and work efficiently with standard transformer implementations.^[1] RoPE belongs to a different family from earlier methods: it does not add anything to the residual stream. Instead, it transforms the query and key vectors right before the attention dot product, so the dot product itself becomes position-aware. The authors contrast "the additive nature of position embedding method adopted in the previous works" with their approach, which "is multiplicative."^[1]

How does RoPE work?

The central idea of RoPE is to encode position by rotating the query and key vectors.^[1] A rotation in two dimensions is a well-understood geometric operation: you take a 2D vector and spin it by some angle. The rotation preserves the vector's magnitude while changing its direction.

RoPE exploits a specific property of rotations: when you compute the dot product between two rotated vectors, the result depends only on the angle difference between the two rotations, not on the individual rotation angles.^[1] If vector A is rotated by angle alpha and vector B is rotated by angle beta, their dot product depends on (alpha - beta), not on alpha and beta individually. Since self-attention is fundamentally a dot product between query and key vectors, this property means that rotating queries and keys by position-dependent angles automatically injects relative position information into the attention scores.^[1]

Step-by-step mechanism

1. Pair up dimensions. Given a query or key vector of even dimension $d$, RoPE divides it into $d/2$ two-dimensional subspaces. In the RoFormer paper the pairs are adjacent coordinates, $(x_1, x_2), (x_3, x_4), \ldots$; many implementations instead pair each coordinate with the one $d/2$ positions later (see Implementation details).^[1][21]

2. Assign rotation frequencies. Indexing the pairs from $i = 0$ to $d/2 - 1$, pair $i$ is assigned the frequency $\theta_i = 10000^{-2i/d}$. (The paper indexes from 1 and writes $\theta_i = 10000^{-2(i-1)/d}$ for $i \in [1, 2, \ldots, d/2]$.)^[1] This follows the same frequency schedule as the original sinusoidal positional encoding.^[3] Lower-indexed pairs rotate at higher frequencies (capturing fine-grained position differences), while higher-indexed pairs rotate at lower frequencies (capturing coarser position information).

3. Compute rotation angles. For a token at position $m$, the rotation angle for pair $i$ is simply $m \theta_i$. Tokens at later positions get rotated by larger angles.

4. Apply 2D rotation to each pair. With coordinates indexed from 0, each pair (x_{2i}, x_{2i+1}) is rotated as:

   • $x'_{2i} = x_{2i} \cos(m \theta_i) - x_{2i+1} \sin(m \theta_i)$
   • $x'_{2i+1} = x_{2i} \sin(m \theta_i) + x_{2i+1} \cos(m \theta_i)$

   This is the standard 2D rotation matrix applied to each pair independently.

5. Compute attention as usual. After rotation, the query and key vectors are used in the standard dot-product attention computation. No other changes to the attention mechanism are needed.

Mathematical formulation

More formally, let $x_m$ be the embedding of the token at position $m$, and let $W_q$ and $W_k$ be the query and key projection matrices. The query and key vectors are:

• $q_m = R(\theta, m) W_q x_m$
• $k_n = R(\theta, n) W_k x_n$

where $R(\theta, m)$ is a block-diagonal rotation matrix (written $R^d_{\Theta,m}$ in the paper). Each $2 \times 2$ block along the diagonal is:

The attention score between positions m and n is:

• $\text{score}(m, n) = q_m^\top k_n = (W_q x_m)^\top R(\theta, m)^\top R(\theta, n) (W_k x_n)$

Because rotation matrices are orthogonal, $R(\theta, m)^\top R(\theta, n) = R(\theta, n - m)$. The score therefore depends on the relative position $(n - m)$ rather than on m and n individually:

• $\text{score}(m, n) = (W_q x_m)^\top R(\theta, n - m) (W_k x_n)$

This is the core mathematical property that makes RoPE work: absolute rotations produce relative position information in the dot product. It corresponds to Equations 14 to 16 of the RoFormer paper, which also notes that multiplying by the sparse rotation matrix directly "is not computationally efficient" and gives an element-wise realization instead.^[1]

What are the properties and advantages of RoPE?

RoPE has several properties that contributed to its widespread adoption:

Relative position through absolute encoding

EleutherAI describes RoPE as a method "that unifies absolute and relative approaches".^[19] Each token is encoded with its absolute position (via the rotation angle), but the attention mechanism naturally computes relative positions (via the angle difference in the dot product). This gives the model both types of information without requiring separate mechanisms.

No additional parameters

RoPE adds zero learnable parameters to the model.^[1] The rotation angles are deterministic functions of position and dimension index. This means RoPE has exactly the same parameter count as a model with no positional encoding at all, and the same count as the original sinusoidal encoding.

Long-term decay, and its limits

Su et al. argue that, with $\theta_i = 10000^{-2i/d}$, RoPE has a "long-term decay" property, "which means the inner-product will decay when the relative position increase," and that this "coincides with the intuition that a pair of tokens with a long relative distance should have less connection."^[1] What the paper derives (its Section 3.4.3) is narrower than that wording suggests. Writing the score as a sum of complex products and applying an Abel transformation, it bounds the score's magnitude by the largest difference between adjacent query-key pair products multiplied by a sum of partial-sum magnitudes $|S_j|$, and shows (its Figure 2) that the average of these magnitudes falls as the relative distance grows.^[1] This is a decaying upper bound, not a guarantee about the score for any particular query and key. Among the paper's stated limitations, the authors write that although the model has this decay property and performs well on long texts, "we have not come up with a faithful explanation."^[1]

Later work questions how much the decay matters. Barbero et al. (ICLR 2025) argue that decay is "unlikely to be the core reason" RoPE is useful, note that such claims "rely on the queries and keys being constant vectors", and report that "there are many situations in which this decay does not occur."^[44] Studying Gemma 7B, they found that the model uses the highest RoPE frequencies to build "positional" attention heads and in general "greatly prefers to use the lowest frequencies", which they suspect carry semantic information.^[44]

Compatibility with KV caching and linear attention

During autoregressive generation, transformers cache the key and value vectors from previous tokens to avoid recomputing them. Because RoPE applies the rotation to query and key vectors before they enter the attention computation, the rotation can be applied once when a token is first processed, and the rotated key can be cached directly. This is straightforward to implement and does not interfere with standard KV caching strategies, which is an important practical consideration for inference efficiency. The exception is a scaling scheme whose factor changes with sequence length: the YaRN authors point out that with Dynamic Scaling "the correct implementation should cache the kv-embeddings before applying rotary position embeddings, as the RoPE of every token changes when s changes."^[8]

A further advantage is compatibility with linear attention. Because RoPE multiplies queries and keys by orthogonal rotation matrices and does not introduce a softmax-dependent bias, it can be combined with kernelized or linear attention variants where additive relative biases are difficult to apply; the RoFormer paper tested it with Performer attention.^[1] The authors specifically highlighted "the capability of equipping the linear self-attention with relative position encoding" as a distinguishing property.^[1]

Natural extension to longer sequences

Because RoPE's rotation angles are continuous functions of position, they are defined for any positive integer position. A model trained with a maximum sequence length of 4,096 will still produce well-defined rotations for position 8,192. Whether the model actually performs well at those positions is another matter: Chen et al. found that when a RoPE model is run directly on context windows longer than it was trained on, "the perplexity may shoot up to very high numbers (i.e., > 10^3), comparable to untrained models."^[6] This gap between a well-defined encoding and poor actual extrapolation is what the context-extension techniques described below address.

How does RoPE differ from other positional encoding methods?

Kazemnejad et al. (NeurIPS 2023), in The Impact of Positional Encoding on Length Generalization in Transformers, trained small decoder-only transformers from scratch on a battery of reasoning and mathematical tasks and found that "NoPE outperforms other explicit positional encoding methods while requiring no additional computation"; they concluded that "explicit position encodings are not essential for decoder-only Transformers to generalize well to longer sequences."^[16] In a 1B-parameter follow-up trained on code with a 1,024-token context, they report that Rotary "fails to generalize as its perplexity explodes" beyond the training length, while NoPE and ALiBi generalize "up to almost twice their training context size", with ALiBi more stable than NoPE at longer lengths.^[16] These results are one reason researchers continue to question whether RoPE is the best choice everywhere, even though it dominates in released models.

Which LLMs use RoPE?

The table lists notable models with the positional details their developers published in a paper, reference code or released configuration file. "Base" is the constant written as 10,000 in the RoFormer frequency formula, called rope_theta in Hugging Face configuration files. Configuration values were read on 31 October 2026. Closed models whose developers have not disclosed their positional encoding are not listed.

The infrastructure ecosystem followed this adoption wave. FlashAttention, vLLM and Hugging Face Transformers all ship RoPE implementations.^[21][54][56] Hugging Face Transformers selects the scaling method from the model configuration; as of version 5.22.1 (30 October 2026) the supported types are default, linear (position interpolation), dynamic (Dynamic NTK), yarn, longrope, llama3 and proportional, stored in a rope_parameters field that also accepts the older rope_scaling key.^[54]

Partial RoPE

Several models rotate only part of each head. GPT-J applies RoPE to 64 of each head's 256 dimensions,^[24] and GPT-NeoX-20B to "the first 25% of embedding vector dimensions"; its authors wrote that their initial experiments indicated this "strikes the best balance of performance and computational efficiency."^[23] The configuration files of Microsoft's Phi-2 (partial_rotary_factor 0.4),^[29] Zhipu's GLM-4.5 (0.5),^[49] Alibaba's Qwen3-Next (0.25)^[48] and Qwen3.6 (0.25),^[51] and MiniMax M2 (64 rotary dimensions out of a 128-dimension head)^[50] use the same idea. Barbero et al. proposed a related variant, p-RoPE, which removes a percentage of the lowest frequencies; they report that this maintained, and on 2-billion-parameter models improved, performance.^[44]

The base parameter and frequency spectrum

The base value 10,000 in $\theta_i = 10000^{-2i/d}$ is a hyperparameter, not a fundamental constant. Su et al. state that they chose it "following Vaswani et al. [2017]", the sinusoidal encoding's schedule.^[1][3] Larger bases produce slower rotations in every dimension pair except the first, which means the same numerical position m corresponds to a smaller phase angle. This effectively spreads positional information over a longer range and is the foundation of NTK-aware context extension.

Many later models raised the base. Code Llama increased it from 10,000 to 1,000,000 for long-context fine-tuning,^[25] Llama 3 increased it to 500,000,^[32] Yi used an adjusted base frequency to support contexts up to 200K,^[28] and Qwen2.5 raised it from 10,000 to 1,000,000 "using the ABF technique (Xiong et al., 2023)" while extending training sequences to 32,768 tokens.^[35] Gemma 3 raised the base from 10,000 to 1,000,000 only on its global self-attention layers and kept 10,000 for its local sliding-window layers.^[55] Barbero et al. suggest their analysis "explains why increasing the maximum RoPE wavelength helps with long-context, e.g. as shown in Llama 3."^[44]

The d/2 frequencies span a wide spectrum. The first few pairs rotate quickly and complete many full revolutions even within a short context window; they encode fine-grained, high-frequency positional information. The last few pairs rotate very slowly. The YaRN authors observe that for dimensions whose wavelength is longer than the maximum training length, the rotation never completes a full turn during pre-training, which they argue lets the network use those dimensions as absolute position information.^[8]

This two-end behavior matters for context extension. Uniform position interpolation, they note, "removes the high frequency components of RoPE", which hurts the model's ability to distinguish nearby tokens, while low-frequency dimensions can be interpolated with less damage. Several context-extension techniques exploit this asymmetry by treating different frequencies differently.^[8]

How is RoPE used to extend context length?

One of RoPE's most active areas of development is extending the context length of pre-trained models beyond their original training length. As noted above, running a RoPE model directly past its training length can make perplexity rise above 1,000.^[6] Research starting in mid-2023 produced a sequence of techniques to extend the effective context window of pretrained RoPE models with little or no fine-tuning.

Position interpolation (PI)

Chen et al. (June 2023, Meta) proposed position interpolation, a simple technique: instead of extrapolating RoPE to positions beyond the training range, interpolate by dividing all position indices by a scaling factor $s$.^[6] If a model was trained with max length 4,096 and you want to extend to 32,768, set $s = 8$ and divide all positions by 8. This maps the extended range [0, 32,768] back into [0, 4,096], where the model has seen positions during training. The authors reported that this "extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps)."^[6] Their theoretical study "shows that the upper bound of interpolation is at least ∼ 600× smaller than that of extrapolation", which they present as evidence of its stability.^[6] Around the same time, the community developer kaiokendev had posted about linearly interpolating RoPE positions for longer sequences, and the community SuperHOT models used a linear scale factor of 4.^[26][7]

NTK-aware scaling

NTK-aware (Neural Tangent Kernel-aware) interpolation was posted by the Reddit user bloc97 on r/LocalLLaMA on 29 June 2023, two days after the PI paper appeared on arXiv.^[7] The insight is that position interpolation treats all frequency dimensions equally, but this is suboptimal. High-frequency dimensions (which capture fine-grained position differences) lose resolution when compressed, while low-frequency dimensions (which capture coarse position information) need compression the most.

NTK-aware scaling addresses this by modifying the base frequency parameter. Instead of scaling position indices, it scales the base value (10,000) in the frequency formula: in the YaRN paper's formulation the new base is $b' = b \cdot s^{d / (d - 2)}$, where $s$ is the desired length-extension factor.^[8] This scales the lowest frequency as much as linear interpolation would while leaving the highest frequency almost unchanged, spreading the interpolation pressure across dimensions.^[8] bloc97 reported that with this change LLaMA 7B did not need fine-tuning to run at a 4,096-token context (twice its training length) with minimal perplexity degradation, and the post's title claimed 8k+ contexts.^[7] The YaRN authors later identified drawbacks: the best base "usually has to be found empirically", some dimensions are pushed to "out-of-bound" values so the scale must be set higher than the intended extension, and fine-tuning with NTK-aware interpolation "yields inferior results to PI."^[8]

According to the YaRN paper, NTK-aware and Dynamic NTK interpolation had already appeared in open models by mid-2023: Code Llama (through its base change to 1,000,000) and Qwen-7B (through Dynamic NTK).^[8][27] A related variant, Dynamic NTK, was posted on r/LocalLLaMA on 30 June 2023 by the user emozilla. It recomputes the scaling as the sequence grows, so the model uses exact positions within its trained length (preserving its trained behavior) and smoothly increases scaling past it.^[26][20] The YaRN authors note that "Dynamic NTK" interpolation "works exceptionally well on models pre-trained on L without any finetuning."^[8]

YaRN (Yet another RoPE extensioN)

Peng et al. (August 2023; published at ICLR 2024) introduced YaRN, which formalized and extended the community NTK-aware methods.^[8] The authors are Bowen Peng and Jeffrey Quesnelle (Nous Research), Honglu Fan (EleutherAI and University of Geneva), and Enrico Shippole.^[8] YaRN's key innovation is recognizing that different frequency ranges require different scaling strategies. It combines two mechanisms:

1. NTK-by-parts (ramp function for dimension-wise interpolation). A ramp function decides whether each dimension is fully interpolated, not interpolated, or partially interpolated, depending on its wavelength relative to the original context length. For LLaMA-family models the authors found $\alpha = 1$, $\beta = 32$ to work well. Low-frequency dimensions receive more interpolation, while high-frequency dimensions receive less.^[8]

2. Temperature scaling for attention logits. YaRN divides the attention logits by a temperature $t$; the authors observed that this has a uniform effect on perplexity across the extended window. For LLaMA and Llama 2 they recommend $\sqrt{1/t} = 0.1 \cdot \ln(s) + 1$, where $s$ is the extension factor, fitted on non-fine-tuned LLaMA models.^[8] Because the scaling can be folded into the precomputed rotary embeddings, it has no runtime overhead.^[8]

The YaRN authors reported that the method needs "10x less tokens and 2.5x less training steps than previous methods" and reaches state-of-the-art context-extension results after fine-tuning on "less than ∼0.1% of the original pre-training data."^[8] They fine-tuned Llama 2 7B and 13B with $s = 16$ (64K context) and $s = 32$ (128K); the $s = 32$ model was trained only on 64K-token data and, according to the authors, extrapolates to 128K.^[8] YaRN was later used in the long-context recipes of DeepSeek-V2 and DeepSeek-V3, applied at inference in Qwen2.5 and Qwen3, and used to extend gpt-oss to 131,072 tokens.^{[17][36][35][41][43]} The released configuration files of Moonshot AI's Kimi K2 and DeepSeek V4-Pro also specify YaRN scaling.^[53][52]

LongRoPE

Ding et al. (February 2024; ICML 2024) at Microsoft introduced LongRoPE, which uses an evolutionary search to find non-uniform rescaling factors for each RoPE dimension and also leaves the first few token positions uninterpolated.^[14] A progressive strategy first fine-tunes a 256K-context model and then applies a second interpolation to reach 2,048K, after which LongRoPE is readjusted at 8K to recover short-context performance.^[14] The authors report extending pretrained LLMs to 2,048k tokens "with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at the original short context window."^[14] The method underpins the 128K-context variants of Phi-3.^[18]

Llama 3.1 frequency-dependent scaling

Meta's reference code for Llama 3.1 rescales RoPE frequencies by wavelength relative to the original 8,192-token context: frequencies whose wavelength is shorter than 2,048 tokens (8,192 divided by a high-frequency factor of 4) are unchanged, those with wavelength longer than 8,192 tokens are divided by a scale factor of 8, and those in between are smoothly interpolated.^[33] Hugging Face exposes this as the llama3 rope type.^[54] The same function, with a configurable high-frequency factor, appears in Meta's Llama 4 reference code.^[33]

Comparison of RoPE scaling methods

Some models adopt the simpler approach of increasing the base during pre-training. Llama 3 used a base of 500,000 (compared with the standard 10,000), which lowers the rotation frequency of every dimension pair except the first.^[32] This approach requires training with the larger base from the start or extensive continued pre-training, and it is often combined with one of the scaling methods above for the final extension step, as in Qwen2.5 and Qwen3.^[35][41]

Implementation details

In practice, RoPE is implemented efficiently without explicitly constructing the full rotation matrices. The rotation of each 2D pair can be computed with simple element-wise multiplications and additions:

q_rotated[2i]   = q[2i] * cos(m * theta_i) - q[2i+1] * sin(m * theta_i)
q_rotated[2i+1] = q[2i] * sin(m * theta_i) + q[2i+1] * cos(m * theta_i)

The cosine and sine values for each position and dimension pair can be precomputed and cached as a lookup table, so the runtime cost of RoPE is a few element-wise multiplications and additions per dimension pair, per token, per layer. This is small compared with the cost of the attention computation itself.

A common compact form, used in modern codebases, expresses the rotation elementwise. Letting cos(m * theta) and sin(m * theta) be vectors broadcast across the head, the rotated query is:

q_rotated = q * cos(m * theta) + rotate_half(q) * sin(m * theta)

where rotate_half splits the vector into two halves (x1, x2) and returns (-x2, x1). This is the form used in the apply_rotary_pos_emb function in the Hugging Face transformers library, in src/transformers/models/llama/modeling_llama.py.^[21] The rotation table is computed by LlamaRotaryEmbedding, whose frequencies come from the scaling functions in modeling_rope_utils.py (linear, dynamic, YaRN, LongRoPE, llama3 and others), selected through the model configuration.^[21][54]

An equivalent formulation restructures the computation using complex number multiplication. If each pair $(x_{2i}, x_{2i+1})$ is treated as a complex number $x_{2i} + i x_{2i+1}$, then the rotation is simply multiplication by the complex number $\cos(m \theta_i) + i \sin(m \theta_i) = e^{i m \theta_i}$.^[1] This view makes the mathematical structure clearer and can be implemented efficiently using complex number support in frameworks like PyTorch; Meta's Llama reference code does this with torch.polar and torch.view_as_complex.^[33]

There are two common layouts for the pairs. Meta's reference code and Hugging Face's GPT-J implementation (rotate_every_two) rotate adjacent, interleaved dimensions, while Hugging Face's GPT-NeoX and Llama implementations use the rotate_half layout, which pairs each dimension with the one half a head away.^[21][33] The two are equivalent up to a permutation of dimensions, but checkpoints converted between formats need a one-time reordering of the query and key projection weights; Hugging Face's Llama conversion script contains a permute function for this purpose.^[21]

RoPE is applied to queries and keys only, not to values, and is applied independently at each attention layer rather than just at the input.^[1] This means the position information is refreshed at every layer, which helps maintain the position signal throughout the network depth. This contrasts with additive methods (sinusoidal, learned embeddings) that apply the position signal only at the input and rely on the network to propagate it through subsequent layers.

Reference implementations include the EleutherAI GPT-NeoX codebase and Su Jianlin's original RoFormer release at ZhuiyiTechnology/roformer on GitHub, a repository created in March 2021.^[22][19]

Connection to relative position bias

RoPE can be viewed as a way to inject the relative-position term $R(\theta, n - m)$ into the attention dot product without adding it as a separate scalar.^[1] T5's relative position bias adds a learned scalar $b(n - m)$ directly to the pre-softmax logit; RoPE multiplies the query and key vectors by complementary rotation matrices so the same relative-position structure emerges from the dot product itself.^[13] The two approaches share a goal but differ in mechanism: T5's bias is additive and learned per relative-distance bucket, while RoPE's rotation is multiplicative and parameter-free.

The long-term decay bound derived by Su et al. is sometimes compared with ALiBi's hand-designed linear penalty.^[1][5] The two are not equivalent: ALiBi subtracts an explicit distance-proportional term from every score, while RoPE's decay is a bound on the score's magnitude that learned queries and keys need not follow.^[5][44]

Extensions and variants

Several extensions to the basic RoPE formulation have been explored:

2D and 3D RoPE extends the rotation approach to two- and three-dimensional position grids, which is useful for vision transformers and other models that operate on spatial data. Instead of a single position index, each token has $(x, y)$ or $(x, y, z)$ coordinates, and the rotation angles are defined over these multi-dimensional positions. Heo et al. (ECCV 2024) studied 2D RoPE in vision transformers and reported that it maintains precision when image resolution is increased at inference, with gains on ImageNet-1k, COCO detection and ADE-20k segmentation.^[45]

Multimodal RoPE (M-RoPE) was introduced with Qwen2-VL in September 2024. Qwen2-VL's vision encoder replaces absolute position embeddings with 2D-RoPE, and its language model decomposes the rotary embedding "into three components: temporal, height, and width." Text tokens use identical IDs for all three components, making M-RoPE "functionally equivalent to 1D-RoPE"; image tokens keep a constant temporal ID with height and width IDs taken from their position in the image; and for video the temporal ID increments for each frame.^[34] Qwen2-VL-7B's configuration assigns 16, 24 and 24 of the 64 frequency pairs to the three components.^[34] Qwen3-VL (September 2025) moved to "Interleaved-MRoPE", described as "full-frequency allocation over time, width, and height", and its documentation advises a smaller YaRN factor than for plain RoPE because Interleaved-MRoPE's position IDs grow more slowly.^[46]

Dynamic NTK adjusts the scaling factor at inference time based on the actual sequence length, rather than using a fixed scaling factor (see above).^[8][20] If the input sequence is shorter than the training length, no scaling is applied; scaling kicks in only when the sequence exceeds the training length.

LongRoPE (Ding et al., 2024) refines dimension-wise scaling by searching for rescaling factors for each dimension, reaching context lengths of up to 2,048K tokens (see above).^[14]

Decoupled RoPE was introduced with DeepSeek-V2's Multi-head Latent Attention. DeepSeek explains that "RoPE is incompatible with low-rank KV compression": a position-dependent rotation between the query and the key up-projection would stop that projection from being absorbed into the query weights, forcing keys for all previous tokens to be recomputed during inference.^[17] Its solution adds extra multi-head queries and a single shared key that carry RoPE, with a per-head RoPE dimension of 64 in DeepSeek-V2; the decoupled key is also cached, and YaRN is applied only to this shared key.^[17] DeepSeek-V3 keeps the design with the same 64-dimension setting,^[36] and the released configurations of Kimi K2 and DeepSeek V4-Pro also list a 64-dimension RoPE component (qk_rope_head_dim).^[53][52] The Multi-head Latent Attention article, linked above, covers the mechanism in detail.

iRoPE and interleaved NoPE layers. Meta's Llama 4 announcement (5 April 2025) describes "interleaved attention layers without positional embeddings" combined with "inference time temperature scaling of attention", which Meta calls the iRoPE architecture, where "i" stands for "interleaved" attention layers, "highlighting the long-term goal of supporting 'infinite' context length", and "RoPE" refers to the rotary embeddings "employed in most layers."^[39] In Meta's reference code a layer is a NoPE layer when its index is a multiple of nope_layer_interval; NoPE layers use a global attention mask while RoPE layers can use chunked local attention, and the temperature scaling is applied only to NoPE layers.^[33] Hugging Face's Llama 4 configuration defaults to a NoPE layer every fourth layer with an attention chunk size of 8,192.^[40] Cohere's Command A (March 2025) uses "interleaved layers of sliding window attention and full attention in 3:1 ratio", with RoPE in each sliding-window layer and no positional embeddings in every full-attention layer, citing Kazemnejad et al. for NoPE.^[37] The design follows Cohere's "Rope to Nope and Back Again" study (Yang et al., January 2025), whose authors report that their hybrid surpasses conventional RoPE-based full-attention transformers on both long- and short-context tasks while being more efficient.^[38] Hugging Face's SmolLM3 (July 2025) adopted the same approach, "selectively removing rotary position embeddings from every 4th layer", which its developers say improved long-context performance without hurting short-context ability in their ablations.^[42]

What are the limitations of RoPE?

While RoPE has proven highly effective, it is not without limitations.

The most cited issue is poor length generalization. Without modification, running a RoPE model past its training length can drive perplexity above 1,000,^[6] and in Kazemnejad et al.'s 1B-parameter experiments Rotary's perplexity "explodes" beyond the training context while NoPE and ALiBi degrade more gradually.^[16] This is the central motivation for PI, NTK-aware scaling, YaRN and LongRoPE.

A second issue is the uneven use of the frequency spectrum. Uniform interpolation removes high-frequency information,^[8] and Barbero et al. prove that the low-frequency bands they believe carry semantic information "cannot be robust over long context", which motivates their p-RoPE variant.^[44]

A third point concerns the distance decay. The decay Su et al. derived is of an upper bound rather than of the attention scores themselves, and Barbero et al. find "many situations in which this decay does not occur", some of which attention heads in Gemma 7B exploit.^[1][44]

A fourth issue is that RoPE is not always optimal for extrapolation. Kazemnejad et al. found that NoPE outperformed RoPE, ALiBi and other explicit encodings on length generalization in small models trained on reasoning tasks, and that ALiBi was the more stable option in their 1B language-modeling runs.^[16] Llama 4, Command A and SmolLM3 all keep RoPE in most layers but remove it from a fraction of them.^[37][39][42]

Practical considerations for context extension

When extending a pretrained RoPE model to a longer context window, the published results suggest the following trade-offs.

• With no fine-tuning, Dynamic NTK and Dynamic YaRN are the options the YaRN authors evaluated; they report that Dynamic NTK "works exceptionally well" on non-fine-tuned models and that Dynamic-YaRN "allows for more than 2x context window extension without any fine-tuning."^[8]
• With a small fine-tuning budget, PI reached 32,768 tokens on LLaMA within 1,000 steps,^[6] and YaRN reports needing fewer tokens and steps than PI-style fine-tuning for comparable extension.^[8]
• For extensions far beyond 128K, LongRoPE reached 2,048K tokens at the cost of an evolutionary search and a progressive fine-tuning schedule.^[14]
• Models trained from the start with a large base (for example Llama 3's 500,000) still often apply a scaling method for the final extension step.^[33][35][41]

The PI authors evaluated passkey retrieval and long-document summarization alongside perplexity.^[6]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). "RoFormer: Enhanced Transformer with Rotary Position Embedding." Submitted to arXiv 20 April 2021; published in *Neurocomputing*, Volume 568, February 2024 (article 127063), doi:10.1016/j.neucom.2023.127063. arxiv.org/...2104.09864
2. ^Su, J. Kexue.fm blog posts: "让研究人员绞尽脑汁的Transformer位置编码" (3 February 2021), kexue.fm/...8130 ; and "Transformer升级之路：2、博采众长的旋转式位置编码" (23 March 2021), kexue.fm/...8265
3. ^1 ^2 ^3Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). "Attention Is All You Need." *NeurIPS 2017*. arxiv.org/...1706.03762
4. ^Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). "Self-Attention with Relative Position Representations." *NAACL 2018*. arxiv.org/...1803.02155
5. ^1 ^2 ^3Press, O., Smith, N. A., & Lewis, M. (2022). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." *ICLR 2022*. arxiv.org/...2108.12409
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). "Extending Context Window of Large Language Models via Positional Interpolation." *arXiv preprint* (submitted 27 June 2023). arxiv.org/...2306.15595
7. ^1 ^2 ^3bloc97 (2023). "NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation." *Reddit r/LocalLLaMA*, 29 June 2023. reddit.com/...led_rope_allows_llama_models_to_have
8. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). "YaRN: Efficient Context Window Extension of Large Language Models." *ICLR 2024*. arxiv.org/...2309.00071
9. ^1 ^2Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). "LLaMA: Open and Efficient Foundation Language Models." *arXiv preprint*. arxiv.org/...2302.13971
10. ^1 ^2Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models." *arXiv preprint*. arxiv.org/...2307.09288
11. ^Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023). "Mistral 7B." *arXiv preprint*. arxiv.org/...2310.06825
12. ^Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context." *ACL 2019*. arxiv.org/...1901.02860
13. ^1 ^2Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." *JMLR 21*. arxiv.org/...1910.10683
14. ^1 ^2 ^3 ^4 ^5Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., & Yang, M. (2024). "LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens." *ICML 2024* (PMLR 235:11091-11104). arxiv.org/...2402.13753
15. ^1 ^2Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). "PaLM: Scaling Language Modeling with Pathways." *arXiv preprint*. arxiv.org/...2204.02311
16. ^1 ^2 ^3 ^4Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., & Reddy, S. (2023). "The Impact of Positional Encoding on Length Generalization in Transformers." *NeurIPS 2023*. arxiv.org/...2305.19466
17. ^1 ^2 ^3 ^4 ^5DeepSeek-AI (2024). "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." *arXiv preprint*. arxiv.org/...2405.04434
18. ^1 ^2Microsoft (2024). "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone." *arXiv preprint*. arxiv.org/...2404.14219
19. ^1 ^2 ^3EleutherAI (20 April 2021). "Rotary Embeddings: A Relative Revolution." *EleutherAI Blog*. blog.eleuther.ai/rotary-embeddings
20. ^1 ^2Fan, H., Peng, B., & Quesnelle, J. (13 November 2023). "Extending the RoPE." *EleutherAI Blog*. blog.eleuther.ai/yarn
21. ^1 ^2 ^3 ^4 ^5 ^6Hugging Face Transformers. `apply_rotary_pos_emb`, `rotate_half` and `LlamaRotaryEmbedding` in `src/transformers/models/llama/modeling_llama.py`; `rotate_every_two` in `models/gptj/modeling_gptj.py`; `permute` in `models/llama/convert_llama_weights_to_hf.py`. github.com/...transformers
22. ^ZhuiyiTechnology. "RoFormer reference implementation." github.com/...roformer
23. ^1 ^2Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., et al. (2022). "GPT-NeoX-20B: An Open-Source Autoregressive Language Model." *arXiv preprint*. arxiv.org/...2204.06745
24. ^1 ^2 ^3EleutherAI. "GPT-J 6B" model card (RoPE applied to 64 of 256 dimensions per head). Hugging Face. huggingface.co/...gpt-j-6b
25. ^1 ^2Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023). "Code Llama: Open Foundation Models for Code." *arXiv preprint*. arxiv.org/...2308.12950
26. ^1 ^2emozilla (2023). "Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning." *Reddit r/LocalLLaMA*, 30 June 2023. reddit.com/...ically_scaled_rope_further_increases
27. ^1 ^2Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023). "Qwen Technical Report." *arXiv preprint*. arxiv.org/...2309.16609
28. ^1 ^201.AI: Young, A., Chen, B., Li, C., et al. (2024). "Yi: Open Foundation Models by 01.AI." *arXiv preprint*. arxiv.org/...2403.04652 ; base value from huggingface.co/...config.json
29. ^1 ^2Microsoft. Phi-2 configuration file (`partial_rotary_factor` 0.4). huggingface.co/...config.json
30. ^1 ^2 ^3Mistral AI. Configuration files for Mistral-7B-v0.1 (`rope_theta` 10,000, `sliding_window` 4,096), Mixtral-8x7B-v0.1 and Mixtral-8x22B-v0.1 (`rope_theta` 1,000,000). huggingface.co/...config.json ; huggingface.co/...config.json ; huggingface.co/...config.json
31. ^1 ^2Gemma Team (2024). "Gemma: Open Models Based on Gemini Research and Technology." *arXiv preprint*. arxiv.org/...2403.08295
32. ^1 ^2 ^3 ^4 ^5Grattafiori, A., Dubey, A., Jauhri, A., et al. (2024). "The Llama 3 Herd of Models." *arXiv preprint*. arxiv.org/...2407.21783
33. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Meta. Llama reference implementations: `apply_scaling` in `models/llama3/model.py` and `models/llama4/model.py`, `nope_layer_interval` in `models/llama4/args.py`. github.com/...llama-models
34. ^1 ^2 ^3 ^4Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., et al. (2024). "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution." *arXiv preprint*. arxiv.org/...2409.12191 ; `mrope_section` from huggingface.co/...config.json
35. ^1 ^2 ^3 ^4 ^5Qwen Team: Yang, A., Yang, B., Zhang, B., et al. (2024). "Qwen2.5 Technical Report." *arXiv preprint*. arxiv.org/...2412.15115
36. ^1 ^2 ^3 ^4DeepSeek-AI (2024). "DeepSeek-V3 Technical Report." *arXiv preprint*. arxiv.org/...2412.19437
37. ^1 ^2 ^3 ^4Cohere (2025). "Command A: An Enterprise-Ready Large Language Model." *arXiv preprint*. arxiv.org/...2504.00698
38. ^Yang, B., Venkitesh, B., Talupuru, D., Lin, H., Cairuz, D., Blunsom, P., & Locatelli, A. (2025). "Rope to Nope and Back Again: A New Hybrid Attention Strategy." *arXiv preprint*. arxiv.org/...2501.18795
39. ^1 ^2 ^3Meta (5 April 2025). "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation." *Meta AI Blog*. ai.meta.com/...llama-4-multimodal-intelligence
40. ^1 ^2 ^3Hugging Face Transformers. `Llama4TextConfig` (`no_rope_layers`, `no_rope_layer_interval`, `attention_chunk_size`) in `src/transformers/models/llama4/configuration_llama4.py`. github.com/...configuration_llama4.py
41. ^1 ^2 ^3 ^4 ^5Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., et al. (2025). "Qwen3 Technical Report." *arXiv preprint*. arxiv.org/...2505.09388
42. ^1 ^2 ^3 ^4Hugging Face (8 July 2025). "SmolLM3: smol, multilingual, long-context reasoner." *Hugging Face Blog*. huggingface.co/...smollm3 ; `no_rope_layer_interval` from huggingface.co/...config.json
43. ^1 ^2 ^3OpenAI (2025). "gpt-oss-120b & gpt-oss-20b Model Card." *arXiv preprint*. arxiv.org/...2508.10925
44. ^1 ^2 ^3 ^4 ^5 ^6 ^7Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., & Veličković, P. (2025). "Round and Round We Go! What makes Rotary Positional Encodings useful?" *ICLR 2025*. arxiv.org/...2410.06205
45. ^Heo, B., Park, S., Han, D., & Yun, S. (2024). "Rotary Position Embedding for Vision Transformer." *ECCV 2024*. arxiv.org/...2403.13298
46. ^Qwen Team. Qwen3-VL README (Interleaved-MRoPE; release of Qwen3-VL-235B-A22B on 23 September 2025). github.com/...Qwen3-VL
47. ^OpenAI. gpt-oss-120b configuration file (`rope_theta` 150,000; YaRN factor 32). huggingface.co/...config.json
48. ^Alibaba Qwen. Qwen3-Next-80B-A3B-Instruct configuration file (`partial_rotary_factor` 0.25). huggingface.co/...config.json
49. ^Zhipu AI (Z.ai). GLM-4.5 configuration file (`partial_rotary_factor` 0.5). huggingface.co/...config.json
50. ^MiniMax. MiniMax-M2 configuration file (`rotary_dim` 64, `head_dim` 128). huggingface.co/...config.json
51. ^1 ^2Alibaba Qwen. Qwen3.6-27B configuration file (`partial_rotary_factor` 0.25, `mrope_interleaved`, `rope_theta` 10,000,000). huggingface.co/...config.json
52. ^1 ^2 ^3DeepSeek. DeepSeek-V4-Pro-0813 configuration file (`qk_rope_head_dim` 64, `rope_theta` 10,000, YaRN factor 16, `original_max_position_embeddings` 65,536). huggingface.co/...config.json
53. ^1 ^2Moonshot AI. Kimi-K2-Instruct configuration file (`qk_rope_head_dim` 64, YaRN factor 32). huggingface.co/...config.json
54. ^1 ^2 ^3 ^4Hugging Face Transformers. `ROPE_INIT_FUNCTIONS` and `RopeParameters` in `src/transformers/modeling_rope_utils.py` (version 5.22.1, 30 October 2026). github.com/...modeling_rope_utils.py
55. ^1 ^2Gemma Team (2025). "Gemma 3 Technical Report." *arXiv preprint*. arxiv.org/...2503.19786
56. ^Rotary embedding code in FlashAttention (`flash_attn/layers/rotary.py`, github.com/...flash-attention) and vLLM (`vllm/model_executor/layers/rotary_embedding`, github.com/...vllm).