Rotary position embeddings (RoPE)

Summary

RoPE encodes token position by rotating each pair of dimensions of the query and key vectors by an angle proportional to position, so that the inner product $Q_{m}^{⊤} K_{n}$ depends only on the relative offset $m - n$ .

Standard absolute position embeddings (sinusoidal in original transformer; learned in BERT/GPT-2) are added to token embeddings at the input. They couple position with content additively and don’t extend cleanly past the training context.

RoPE (Su et al., 2021) is a multiplicative scheme applied inside attention. It became the default in modern decoder LLMs: Llama 1/2/3, Mistral, Qwen, DeepSeek, GPT-NeoX. Its rotation formula has no learned position-table cutoff, although using it far beyond the trained context can still degrade quality. Context-extension methods such as NTK-aware scaling, YaRN, and position interpolation modify the frequencies or the positions supplied to RoPE.

The mechanism

Split the head dimension $d$ into $d /2$ pairs. For each pair $i \in {0, \dots, d /2 - 1}$ pick a frequency $θ_{i} = 1000 0^{- 2 i / d}$ (same base as sinusoidal). For a token at position $m$ , rotate the $i$ -th 2D pair by angle $m θ_{i}$ :

(q_{2 i}^{'} q_{2 i + 1}^{'}) = (cos (m θ_{i}) sin (m θ_{i}) - sin (m θ_{i}) cos (m θ_{i})) (q_{2 i} q_{2 i + 1})

Apply the same rotation to keys (with their position $n$ ). The inner product satisfies $⟨ Q_{m}^{'}, K_{n}^{'} ⟩ = f (Q_{m}, K_{n}, m - n)$ . Depends only on the relative offset.

In code, RoPE is implemented as elementwise multiplies with precomputed cos and sin tables; no extra parameters.

Learning objective

A shared position shift preserves the relative rotary angle, not every model output

Derivation

compare the two circles. Adding three to both token positions turns (1, 3) into (4, 6) and rotates both vectors by the same extra angle. Their absolute directions change, but their 2θ separation does not. Algebraically, RₘᵀRₙ = Rₙ₋ₘ, so this pair's contribution to the attention score depends on n − m. Solid Q and dashed K arrows keep the distinction visible without color. Original schematic, checked against the RoFormer paper.

Check: Why does shifting both positions by three preserve this positional dot-product contribution without asserting identical whole-model outputs?

Read it this way: A common 45-degree rotation preserves the 90-degree angle between these query/key directions. It does not imply every model output is shift-invariant.
Does the ninety-degree angle change when both arrows rotate forty-five degrees?

Why it works

Relative: attention scores depend on $m - n$ , not absolute positions, matching what attention should care about.
Distance-sensitive: high-frequency pairs ( $i$ small) rotate fast. Each pair’s contribution oscillates with distance; across many frequencies, their combined relative-position signal tends to weaken as distance grows.
Length-flexible formula: rotations are defined at any position, so RoPE has no learned-table cutoff, but reliable use beyond the training range still requires care.

Context extension

To run a RoPE model past its training length:

Position interpolation (Chen et al., 2023): linearly compress positions so the new max length maps to the original training range.
NTK-aware scaling: change the rotary frequency grid, for example through a base adjustment. Variants modify frequency bands differently; a changed base does not by itself establish reliable long-context behavior.
YaRN (Peng et al., 2023): per-frequency interpolation tuned by training length statistics.

They alter the position-to-angle mapping without changing the attention architecture.

Common pitfalls

Changing the checkpoint’s construction. In standard RoPE here, rotate Q and K, not V. A different position mechanism requires its own compatible training and implementation contract.
Confusing with ALiBi. ALiBi adds a fixed slope to attention scores; RoPE rotates Q/K. Both encode relative position but are different mechanisms.
Assuming the formula proves extrapolation. Evaluation beyond the trained range must test position mapping, actual context use, and serving/cache limits. Degradation can occur, but there is no universal failure point shared by all RoPE checkpoints. See the position-mechanism comparison.