Skip to content
mlmentorship

Pipeline parallelism

Split the model across GPUs by layer; pipeline mini-batches through the stages. The way to scale across slow interconnects when TP isn't viable.

Published · 8 min read ·Role-specific ·Advanced

Summary

Pipeline parallelism (PP) splits a model along its depth: GPU 0 holds layers 1–8, GPU 1 holds layers 9–16, etc. A mini-batch is divided into smaller micro-batches that flow through the stages so that GPU 0 starts processing micro-batch 2 while GPU 1 processes micro-batch 1, achieving parallel utilization despite the sequential layer dependency.

For large models that do not fit on one accelerator, tensor parallelism is often placed on the fastest links because it communicates every layer. Pipeline parallelism sends activations only between adjacent stages, so it can often use slower links more efficiently. Tensor parallelism can cross nodes when the network and workload support it. The choice follows measured communication cost, not a fixed node boundary.

The basic idea

Without micro-batches, naive pipeline:

GPU 0: forward layer 1-8 ────────────  →  backward layer 1-8 ──── 
GPU 1: ───────────  forward 9-16 ────  →  backward 9-16 ───────
GPU 2: ─────────────────  forward 17-24 → backward 17-24 ─

Most GPUs are idle most of the time. The pipeline bubble.

With micro-batches, the bubble shrinks:

GPU 0: f1  f2  f3  f4 ─────────────────────  b4  b3  b2  b1
GPU 1: ─── f1  f2  f3  f4 ─────────  b4  b3  b2  b1 ───────
GPU 2: ─────── f1  f2  f3  f4  b4  b3  b2  b1 ─────────────

Bubble fraction .

Learning objective · P = 3 stages · M = 4 micro-batches

Idle stage-slots measure unused capacity, not extra elapsed time

Idle stage-slots are unused capacity, not extra elapsed timeThree microbatches cross three stages in five time slots. Six idle stage-slots occur within that same elapsed schedule.Follow each microbatch through thediagonal stage waveABC---ABC---ABCFive elapsed slots; nine busy and sixidle stage-slots.Idle stage capacity is already insidethe same wall clock.
Derivation
Three-stage pipeline schedule for four micro-batches A grid has three stage rows and six time columns. Stage 1 processes micro-batches 1 through 4 in time slots 1 through 4, then has two idle slots. Stage 2 begins with one idle slot, processes micro-batches 1 through 4 in slots 2 through 5, then has one idle slot. Stage 3 begins with two idle slots and processes micro-batches 1 through 4 in slots 3 through 6. There are 12 active cells and 6 dotted idle cells, so the visible bubble is 6 of 18 stage-slots, or one third. STAGE TIME SLOT → 1 2 3 4 5 6 S1 S2 S3 early middle late F1 F2 F3 F4 F1 F2 F3 F4 F1 F2 F3 F4 · idle · · idle · · idle · · idle · · idle · · idle · F1 advances FILL · 3 idle slots FULL · all stages busy DRAIN · 3 idle slots 12 active + 6 idle = 18 stage-slots bubble = 6 / 18 = 1 / 3 = (P − 1) / (M + P − 1)
follow F1 diagonally: each micro-batch must finish an earlier stage before entering the next, so later stages idle while the pipeline fills and earlier stages idle while it drains. Here 12 useful cells and 6 dotted idle cells occupy 18 stage-slots, making the bubble fraction 6/18 = 1/3. Increasing M extends the full middle without adding more fill or drain depth.
Check: Why is the bubble six of eighteen stage-slots, not six extra elapsed slots?
Read it this way: Three microbatches cross three stages in five time slots. Six idle stage-slots occur within that same elapsed schedule.
Should six idle stage-slots be added to the five-slot wall clock?

GPipe vs. 1F1B vs. interleaved

  • GPipe (Huang et al., 2018): all forwards then all backwards. Bubble fraction high; activation memory high (must store all forwards).
  • 1F1B (one forward, one backward; PipeDream): start backward as soon as the first micro-batch reaches the last stage. For fixed stages and micro-batches, it has the same basic fill/drain bubble as GPipe but lowers peak activation memory by releasing each micro-batch sooner.
  • Interleaved 1F1B (Megatron): each GPU holds non-contiguous chunks of layers (e.g., layers 1-2 and 9-10) so the bubble shrinks further.
  • Zero Bubble Pipeline (recent): split backward into weight-grad and input-grad parts to fill almost all bubbles.

Large training systems may use interleaved 1F1B or zero-bubble schedules when simple schedules leave too much idle time.

Cost model

For a model with layers split across stages and micro-batches:

  • Bubble: of total time. Minimize by increasing .
  • Communication per micro-batch: send activations between adjacent stages. Cost scales with micro-batch × sequence × hidden size. Whether this is cheaper than another layout depends on message frequency, dtype, topology, and overlap.
  • Activation memory per stage: in 1F1B, micro-batches’ worth of activations.

When PP wins

  • Cross-node scaling with slow interconnect.
  • Very deep models where one stage easily fits on a node.
  • Frontier training combining 3D parallelism (DP + TP + PP).

When PP loses

  • Small models that fit on one node. TP within node + DP across nodes is simpler.
  • Few micro-batches in a step. The bubble dominates.
  • Workloads with very different per-layer compute. Load imbalance creates idle GPUs.

3D parallelism

One common large-model layout combines:

  • Tensor parallelism on the fastest useful links.
  • Pipeline parallelism across balanced groups of layers.
  • Data parallelism or FSDP across the remaining replica dimension.

The degrees must multiply to the available device count, but that arithmetic does not select the layout. Memory, global batch, topology, and measured scaling select it.

Common pitfalls

  • Few micro-batches create a large bubble. More micro-batches improve utilization, but they interact with activation memory and the global batch.
  • Imbalanced stages. Embeddings, output layers, and uneven blocks can make one stage slower than the others.
  • Forgetting activation memory. The schedule controls how many micro-batches’ activations remain live. Combine PP with activation checkpointing when needed.
  • Treating PP as the same as TP. They split different axes and communicate at different points. Compare their actual activation messages and schedule overhead.
  • Using a fixed topology rule. A node boundary is not a law. Measure the links and collectives on the target cluster.
  • Skipping schedule comparison at many stages. Interleaving can reduce bubbles, but it changes communication and model placement. Measure both schedules.

Found an error or stale claim? Report a correction or send a private note.