Rotary Position Embeddings Rotate Pairs of Coordinates

Move the same two token representations from positions 12 and 15 to positions 100 and 103. In rotary position embedding (RoPE), their positional contribution to the query-key score is the same: both pairs are three positions apart. That property follows from rotating the query and key vectors before their dot product, rather than attaching a separate position label to the final score.

The RoFormer paper starts with the problem that self-attention needs order information. A token’s content is projected into a query and a key. RoPE splits each projected vector into two-dimensional coordinate pairs. For one pair (x, y) at position p, it applies an ordinary rotation through angle pθ:

x' = x cos(pθ) - y sin(pθ)
y' = x sin(pθ) + y cos(pθ)

Each pair uses its own frequency θ. In the paper’s construction, frequencies span several scales, so different coordinate pairs change phase at different rates as positions advance. Queries and keys receive the same positional rotation rule; the original RoPE construction leaves values outside that rotation. A rotation preserves the length of each pair. It changes the direction used in the attention score without adding a learned position vector or changing that pair’s magnitude.

Why does a rotation applied at an absolute position produce a relative-position effect? Let R(p) mean the block of pairwise rotations for position p. A query from position m and a key from position n contribute (R(m)q) · (R(n)k). Rotations are orthogonal, so this is q · R(n-m)k. The score’s positional term depends on the displacement n-m. Moving both representations together by the same number of positions keeps that displacement unchanged. The content vectors still matter: two different tokens three positions apart need not receive the same score.

Take a single coordinate pair to see the geometry. If both unrotated vectors point along the positive horizontal axis, their rotated dot product is proportional to cos((n-m)θ). That is an illustration of one pair, not the whole model. Real attention adds contributions from many pairs, mixes them with learned query and key projections, scales the score, and normalizes scores across available keys. RoPE makes relative distance available inside those content-dependent scores; it does not prescribe which token the model must attend to.

This distinction matters when an AI coding assistant processes a long repository excerpt or a voice system carries a transcript across turns. If cached keys were rotated for their original token positions, a new query must use its actual position in the same sequence. Restarting position numbering for a later chunk changes the relative rotations and therefore the scores. When implementing chunked inference or a key-value cache, keep the model’s position convention consistent across prefill and generation. The formula also explains why adding new positions is mechanically possible: the rotations can be evaluated beyond the sequence lengths seen during training.

Mechanical possibility does not establish useful long-context behavior. The model learned its projections and attention patterns on a training distribution. At unfamiliar distances, the phases may revisit similar orientations, learned behaviors may fail to transfer, and attention still has to compete among more candidate tokens. The paper reports long-text experiments and a theoretical account of some properties, while its limitations acknowledge that the observed long-text advantage is not fully explained. It does not establish that extending a model’s context limit by changing a setting preserves retrieval accuracy, reasoning quality, or memory use.

There is a second constraint outside positional math: ordinary full attention still considers token pairs, and serving longer prompts consumes more attention work and cache storage. RoPE changes how position enters the score. It does not solve those resource costs. A context extension technique may also alter the frequencies or position mapping; such a change should be treated as a model-specific intervention rather than an automatic consequence of the original formula.

For an implementation decision, first verify the model’s coordinate pairing and position-index convention, then test a long-input task that matters to users: finding a symbol in a repository excerpt, retrieving a fact from an early transcript turn, or preserving a constraint buried in a document. Measure accuracy by distance as well as total prompt length. The rotation identity tells you what positional relationship the score can represent; the evaluation tells you whether this model uses it reliably.