Prefill vs. decode: the two phases of LLM inference

Summary

Prefill processes available prompt positions together; decode processes a newly sampled input after its predecessor becomes available. Compare available rows per weight read before naming the bottleneck.

Consider a $1024 \times 1024$ BF16 weight matrix W: $2, 097, 152$ bytes, or 2 MiB. Assume each invocation streams W once, with no inter-call residency. Ignore input/output bytes, KV traffic and kernel overhead for this accounting exercise.

How do four simultaneous rows differ from four serial one-row invocations?

Same arithmetic; different ideal weight traffic

Shared weight loads can serve the same arithmetic across several tokens

Derivation

W has shape 1024 by 1024. The serial calls represent four subsequent decode steps, not four total output tokens.

Prefill · one parallel pass

Four prompt-token rows are available together.

HBM supplies	Matrix work
weight tile W load for this tile	rows 1 · 2 · 3 · 4 reuse W across prompt rows

Weight-only accounting: 8,388,608 FLOPs / 2,097,152 bytes = 4 FLOPs/byte.

Decode · four serial passes

At batch 1, only one new-token row exists per step.

Decode invocation	HBM and matrix work
1	load W → row 1
2	load W again → row 2
3	load W again → row 3
4	load W again → row 4

Weight-only accounting: 8,388,608 FLOPs / 8,388,608 bytes = 1 FLOP/byte.

four parallel rows reuse each loaded weight. Serial one-row calls repeat the stream under the stated residency assumption; actual traffic requires measurement.

Check: Why is arithmetic identical while the illustrated serial path loads four times as many weight bytes?

Read it this way: Prefill or batching can reuse loaded weights across tokens. Decode still performs token-specific computation and reads its growing cache.
Which per-token work remains after the weight load is shared?

One row costs $2 (1024)^{2} = 2, 097, 152$ FLOPs, counting a multiply-add as two operations. Both schedules perform $8, 388, 608$ FLOPs; their ideal weight traffic differs fourfold. Neither ratio alone establishes a hardware bottleneck.

Prefill

For prompt length $P$ , compute K/V and hidden outputs for available positions with the required causal mask. The final prompt position supplies the first output-token distribution.

Using $N$ participating dense weights, the dense work is approximately $2 P N$ FLOPs. Add attention work $O (L P^{2} d)$ for $L$ layers and hidden width $d$ . More rows can improve reuse; they need not produce square matrices or saturate the device.

Large efficient shapes can be compute-bound; short prompts can be memory-bound or launch-bound.

Decode

Each subsequent invocation processes the sampled token, appends its K/V, and predicts the next token. For batch size $B$ , dense work is approximately $2 B N$ , plus context-dependent attention and KV traffic.

Small-batch weight streaming can dominate, but long-context KV reads, compute and launches can become limiting. See the KV state ledger and resource lower bounds.

Latency metrics

TTFT measures arrival-to-first-token latency, including queueing, scheduling, prompt processing, sampling and transport as applicable. Inter-token intervals describe later outputs; TPOT is often their average, not a constant service time.

For $T \geq 1$ outputs, end-to-end latency is TTFT plus the $T - 1$ subsequent intervals. A separate hypothetical request with TTFT 100 ms and three 20-ms intervals finishes four outputs in $100 + 3 (20) = 160$ ms.

Change the rows or arrival clock

1. Sixteen requests are ready

Picture and change: batch sixteen independent decode rows while retaining separate positions, masks, KV and sampler states.

Check: weight-only intensity becomes 16 FLOPs/byte, not a guaranteed sixteenfold speedup.

Use when: measurements identify weight traffic as a limiting term.

2. Add 20 ms of queueing

Picture and change: measure from arrival, not GPU launch. If prefills block decoding, bound chunks and track the prompt cursor with KV.

Check: TTFT becomes 120 ms and four-output latency 180 ms, with other timings unchanged.

Use when: load or long prompts violate first-token or inter-token targets.

Failure and transfer

Calling every prefill compute-bound misses launch-bound short prompts; calling every decode weight-bound misses other traffic and compute.

How many subsequent decode invocations normally produce four outputs after prefill supplies the first?

Check your answer

Three. The first sampled output enters KV only when processed by a subsequent invocation, not when sampled.

What follows from the asymmetry

Optimization	Conditional effect, not a universal phase ranking
Larger batches	Reuse weights across rows; queueing, compute and KV can limit benefits
FlashAttention	Reduce attention traffic; actual gain depends on shape and implementation
Weight quantization	Reduce weight bytes; kernel support, conversion and errors matter
KV quantization	Change cache capacity/traffic, including writes; metadata and accuracy remain costs
Speculation	Spend draft/verification work to reduce sequential target calls when accepted
GQA/MQA	Change architecture and KV size, requiring compatible trained weights
Continuous batching	Reuse completed request positions under token, memory and fairness constraints

Common pitfalls

Measure full workloads, not a single inference-cost number. Neither output length alone nor phase name determines the dominant latency term. Speculation can lose on short outputs when overhead exceeds saved work.

References: Pope et al., Orca, SARATHI.

Continuous batching covers admission and per-request state. Speculative decoding covers draft verification.