Positional encoding: where does position enter attention?

Summary

Position can change embeddings, query/key geometry, or scores before softmax. Those are different computations, not interchangeable ways to relabel a checkpoint. Unmasked position-free self-attention is permutation-equivariant: permuting input rows permutes output rows. A fixed causal mask already restricts positions asymmetrically.

Task: query position $i=6$ may read keys $j=[2,4,6]$. Hold scaled content scores at zero and use causal ALiBi slope $s_h=0.25$. Compute the attention probabilities, then change the slope and permitted directions.

The lineup

The pipeline is embeddings, Q/K projections, dot product, bias/mask, then softmax. Locate each method at its actual stage.

Sinusoidal (original Transformer, 2017)

Add fixed absolute-position vectors before the projections:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Defined values at unseen positions do not guarantee useful extrapolation.

Learned absolute positional embeddings

Add a learned table row for each supported position, as in BERT or GPT-2. Unseen indices need an explicit adaptation, not an undefined table lookup.

Relative positional encoding

T5-style relative bucket biases enter scores; bucket definitions are part of the model. Shaw and Transformer-XL use other relative constructions, not the same lookup formula.

RoPE (Rotary Position Embeddings)

Rotate query/key coordinate pairs before their dot product, rather than adding an embedding vector.

For positions m and n, use their specified block-diagonal rotations:

Q' = R_m * Q   (Q rotated by query position)
K' = R_n * K   (K rotated by key position)

The rotation derivation proves the common-shift relationship for fixed content vectors. The figure retains positions $(1,3)$ and $(4,6)$; their relative offsets agree.

ALiBi (Attention with Linear Biases)

Add a distance bias after the scaled content score:

score_h(i, j) = Q_i K_j^T / sqrt(d) - s_h * (i - j)

Here $d$ is head width, $s_h>0$ is a fixed head-specific slope, and allowed causal keys satisfy $j\le i$. Forbidden scores still require a separate mask.

Compute and change the row

For the first row, exponentials are $[0.367879,0.606531,1]$, summing to $1.974410$. The newest key is $e^1$ times as likely as the oldest because their content scores match. Shifting query to $9$ and keys to $[5,7,9]$ preserves distances and probabilities for unchanged content.

New task: strengthen recency preference. Double the slope and recompute softmax, as the second row shows. Use when: deliberately adapting head distance preferences; changing deployed slopes changes the model.

New task: allow an encoder query at $6$ to read key $8$. Use a bidirectional mask and the specified absolute-distance bias. Check: slope $0.25$ gives $-0.25|6-8|=-0.5$, not $+0.5$ from misusing signed causal distance. Use when: available source positions lie on both sides.

Position-free / implicit position

A causal mask or recurrence can impose order without adding a positional vector. Mixture-of-experts routing alone does not supply sequence position.

What an interviewer expects you to say

Separate position representation, visibility, and context capacity. Failure: a fixed slope is not a causal mask; unmasked future keys can receive an unintended positive signed-distance bias.

Reconstruct: can RoPE replace ALiBi unchanged because both use relative positions?