Rotary position embeddings (RoPE)

Summary

RoPE encodes token position by rotating each pair of dimensions of the query and key vectors by an angle proportional to position, so that the inner product $Q_{m}^{⊤} K_{n}$ depends only on the relative offset $m - n$ .

Standard absolute position embeddings (sinusoidal in original transformer; learned in BERT/GPT-2) are added to token embeddings at the input. They couple position with content additively and don’t extend cleanly past the training context.

RoPE (Su et al., 2021) is a multiplicative scheme applied inside attention. It became the default in modern decoder LLMs: Llama 1/2/3, Mistral, Qwen, DeepSeek, GPT-NeoX. Its rotation formula has no learned position-table cutoff, although using it far beyond the trained context can still degrade quality. Context-extension methods such as NTK-aware scaling, YaRN, and position interpolation modify the frequencies or the positions supplied to RoPE.

The mechanism

Split the head dimension $d$ into $d /2$ pairs. For each pair $i \in {0, \dots, d /2 - 1}$ pick a frequency $θ_{i} = 1000 0^{- 2 i / d}$ (same base as sinusoidal). For a token at position $m$ , rotate the $i$ -th 2D pair by angle $m θ_{i}$ :

(q_{2 i}^{'} q_{2 i + 1}^{'}) = (cos (m θ_{i}) sin (m θ_{i}) - sin (m θ_{i}) cos (m θ_{i})) (q_{2 i} q_{2 i + 1})

Apply the same rotation to keys (with their position $n$ ). The inner product satisfies $⟨ Q_{m}^{'}, K_{n}^{'} ⟩ = f (Q_{m}, K_{n}, m - n)$ . Depends only on the relative offset.

In code, RoPE is implemented as elementwise multiplies with precomputed cos and sin tables; no extra parameters.

Learning objective

Why do absolute rotations produce a relative attention score?

Read it this way: compare the two circles. Adding three to both token positions turns (1, 3) into (4, 6) and rotates both vectors by the same extra angle. Their absolute directions change, but their 2θ separation does not. Algebraically, RₘᵀRₙ = Rₙ₋ₘ, so this pair's contribution to the attention score depends on n − m. Solid Q and dashed K arrows keep the distinction visible without color. Original schematic, checked against the RoFormer paper.

Why it works

Relative: attention scores depend on $m - n$ , not absolute positions, matching what attention should care about.
Distance-sensitive: high-frequency pairs ( $i$ small) rotate fast. Each pair’s contribution oscillates with distance; across many frequencies, their combined relative-position signal tends to weaken as distance grows.
Length-flexible formula: rotations are defined at any position, so RoPE has no learned-table cutoff, but reliable use beyond the training range still requires care.

Context extension

To run a RoPE model past its training length:

Position interpolation (Chen et al., 2023): linearly compress positions so the new max length maps to the original training range.
NTK-aware scaling: increase the RoPE base $10000$ to a larger value so high-frequency components don’t alias.
YaRN (Peng et al., 2023): per-frequency interpolation tuned by training length statistics.

They alter the position-to-angle mapping without changing the attention architecture.

Common pitfalls

Applying RoPE to V. Only Q and K are rotated; V is not.
Confusing with ALiBi. ALiBi adds a fixed slope to attention scores; RoPE rotates Q/K. Both encode relative position but are different mechanisms.
Forgetting the base when extending context. Naively running a 4K model at 32K without scaling produces garbage past 4K.