Skip to content
mlmentorship

Continuous batching for LLM serving

Let new requests join an in-flight batch at every decode step instead of waiting for the slowest one. The other half of why vLLM is fast.

Published · 5 min read ·Role-specific ·Advanced

Visual quick review

Visual first · depth when needed

Trace how iteration-level scheduling replaces a completed request with a queued request at the next decode step while unfinished requests remain active.

Preparing the visual…

Summary

Continuous batching (a.k.a. iteration-level scheduling) processes a batch one decode step at a time and lets new requests enter the batch as soon as another request finishes, instead of waiting for the entire static batch to complete.

Small-batch LLM decoding is usually memory-bound because each step reads model weights for little arithmetic. Adding requests can amortize those weight reads and improve throughput. Iteration time eventually grows as matrix compute, KV-cache traffic, or scheduler overhead becomes limiting.

With static batching, you wait for the longest request in the batch before reusing GPU. If one request generates 1000 tokens and another generates 50, the second request’s GPU slot sits idle for 950 steps.

Continuous batching keeps more useful work on the GPU. Combined with paged KV allocation, it is a common design in modern LLM servers. The gain over static batching depends on arrival rate, length distribution, memory capacity, scheduler policy, and latency target.

Learning objective

What changes when the scheduler can rebuild the batch after every decode step?

Static batching leaves a finished request's slot idle Across five decode steps, request A occupies slot one for all five steps. Request B occupies slot two for steps one and two, then slot two is idle for steps three through five. Queued request C waits because the static batch cannot change until A also finishes. Seven of ten slot positions do useful work. STATIC · MEMBERSHIP FIXED 1 2 3 4 5 slot 1 slot 2 A A A A A B B idle idle idle C waits until A finishes after step 5. USEFUL SLOTS: 7 / 10
Continuous batching replaces B with queued request C at the next iteration Across the same five decode steps, request A occupies slot one throughout. Request B occupies slot two for steps one and two. After B completes, the scheduler admits queued request C into slot two for steps three through five while A continues. All ten slot positions do useful work. CONTINUOUS · REBUILD EACH STEP 1 2 3 4 5 slot 1 slot 2 A A A A A B B C C C C admitted for step 3 USEFUL SLOTS: 10 / 10
Read it this way: B finishes after step 2 in both schedules. Static batching strands that slot until A finishes; continuous batching rebuilds the active set and admits queued C for step 3 while A keeps decoding.

The mechanism

Each “step” of the server runs one forward pass for the current batch:

  1. Maintain a queue of pending requests and a set of active (in-flight) requests.
  2. Each step, build a batch of (a) one decode token from each active request whose KV cache exists and (b) prefill tokens for newly admitted requests.
  3. Run one forward pass. Update each active request’s KV cache.
  4. For requests that hit EOS or max_tokens, mark complete and free their KV blocks.
  5. Admit new requests from the queue if there is enough free KV-cache capacity.

This requires the attention kernel to handle variable per-request lengths in the same batch (cu_seqlens-style cumulative offsets) and a non-contiguous KV cache (PagedAttention).

Prefill vs. decode

Two phases with very different cost profiles:

  • Prefill: process the full prompt in one parallel matmul. Compute-bound; high arithmetic intensity.
  • Decode: one new token per step per request. Memory-bound; benefits hugely from batching.

Most servers either alternate prefill and decode steps or interleave (chunked prefill, Patel et al., 2023) so neither phase starves the other.

Tradeoffs

  • Throughput vs. latency: larger batches mean higher tokens/sec across the server but slightly higher per-request latency. SLO-aware servers cap batch size or fragment large prefills.
  • Memory pressure: continuous batching is throughput-limited by KV-cache memory, not by compute. PagedAttention removes most fragmentation; GQA / MQA shrink per-request cache.
  • Fairness: a long-context request consumes more KV per step. Without admission control, it can starve short requests.

Common pitfalls

  • Profiling decode without batching. Single-request decode benchmarks dramatically understate server throughput.
  • Confusing batch size with sequence length. Batch size grows the number of concurrent requests; longer sequences grow per-request KV.
  • Assuming static batching is fine for production. It almost never is. The GPU sits idle whenever any request finishes.

Found an error or stale claim? Report a correction or send a private note.