Self-attention vs cross-attention

Summary

Self-attention computes $softmax (Q K^{⊤} / d) V$ where $Q, K, V$ are all derived from the same sequence. Cross-attention uses $Q$ from one sequence and $K, V$ from another. Same kernel, different routing.

This input choice separates encoder-only models (BERT), decoder-only models (GPT), and encoder-decoder models (T5, the original Transformer, and Whisper). Multimodal architectures also use cross-attention to connect image, text, or audio representations.

If you can write the matrix multiplications and explain why a layer uses one form, you understand the attention structure of common transformer architectures.

Self-attention

Inputs: a single sequence $X \in R^{n \times d}$ .

Q = X W_{Q}, K = X W_{K}, V = X W_{V},

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{h}}) V .

Each token attends to every other token (subject to masking). Used in:

BERT encoder layers: bidirectional self-attention.
GPT decoder layers: causal self-attention. The mask zeros out positions $j > i$ so token $i$ cannot attend to future tokens.

Cross-attention

Inputs: a query sequence $X$ and a key-value sequence $Y$ .

Q = X W_{Q}, K = Y W_{K}, V = Y W_{V} .

Same softmax, same scaling. The shape of the attention matrix is now $∣ X ∣ \times ∣ Y ∣$ .

Used wherever the model needs to “look up” information from a different source:

Encoder-decoder transformers: decoder layers cross-attend to the encoder output. Translation, summarization, speech-to-text.
Diffusion models with text conditioning: image-side latents cross-attend to text embeddings. Stable Diffusion, DiT.
Perceiver / Q-Former: a small set of learned latent queries cross-attend to a large input (image patches, audio frames) to compress it.
RAG architectures with separate memory: model states can cross-attend to retrieved document representations. Many decoder-only RAG systems instead concatenate retrieved text into the prompt and use self-attention.

Learning objective

Queries determine output rows even when keys and values come from another sequence

Derivation

follow the projection inputs, not the operation name. In self-attention, one sequence X fans out to Q, K, and V, so three query tokens scored against three key tokens form a square 3 × 3 matrix. In cross-attention, X supplies two queries while Y supplies three key-value pairs, so the same kernel forms a rectangular 2 × 3 matrix and returns one row per X query. Source labels, arrows, and matrix dimensions carry the distinction without color. Original schematic, checked against Attention Is All You Need.

Check: Why does the cross-attention example return two rows despite reading three memory tokens?

Read it this way: Two query positions attending to three source positions produce two output rows. The number of source keys does not create an extra output query.
Do three source keys create a third output when there are only two queries?

Where each lives in a transformer block

Encoder block (BERT, T5 encoder):

Self-attention.
FFN.

Decoder block (GPT):

Causal self-attention.
FFN.

Encoder-decoder block (T5 decoder, original Transformer decoder):

Causal self-attention.
Cross-attention to encoder output.
FFN.

The decoder reads its own past tokens (self-attention) and the encoder’s output (cross-attention) at every layer.

Tradeoffs

Compute: self-attention is $O (n^{2} d)$ . Cross-attention is $O (n_{Q} \cdot n_{K V} \cdot d)$ , which can be much cheaper if $n_{K V}$ is small (compressed conditioning) or much larger (cross-attending to a long context).
KV-cache: at encoder-decoder inference, the decoder can cache both forms. Cross-attention K/V are computed once from the fixed encoder output and reused for every decoded token; self-attention K/V grow with the decoded sequence.

Variants

Masked cross-attention: padding masks hide invalid K/V positions. A triangular mask is meaningful only when query and K/V positions share an ordered alignment; separate source and target sequences do not imply one automatically.
Cross-attention with caching: precompute $K, V$ from a fixed conditioning sequence (system prompt, retrieved docs) and reuse across decoding steps.
Asymmetric cross-attention: in Perceiver, the queries are a small learned set (e.g. 256), the K/V are massive (e.g. all image patches). The model compresses high-dim input into a fixed-size latent.

Common pitfalls

Calling decoder self-attention “cross-attention.” They are different. Self-attention reads from the same sequence (the previously generated tokens); cross-attention reads from another sequence (encoder output, retrieved docs).
Forgetting that decoder-only LLMs do not use cross-attention. Their conditioning is the prompt prefix, attended via self-attention, not a separate cross-attention path.
Conflating attention masks with attention types. The mask shape differs (causal mask is square and triangular; cross-attention is rectangular and usually unmasked) but the operation is the same.