Skip to content
mlmentorship

All-reduce and other collectives

The communication primitives behind every distributed training job. All-reduce, all-gather, reduce-scatter, broadcast. What they do, costs, and when each is used.

Published · 8 min read ·Role-specific ·Advanced

Visual quick review

Visual first · depth when needed

Trace one tensor chunk through the three reduce-scatter hops of a four-rank ring, then connect both three-step phases to the per-rank traffic formula.

Preparing the visual…

Summary

A collective communication operation is a coordinated message-passing primitive across a group of processes. The five most common in deep learning: broadcast (one-to-all copy), reduce (all-to-one sum), all-reduce (all-to-all sum into all), all-gather (concatenate everyone’s data into all), reduce-scatter (sum-then-shard).

Every distributed training run is built on these primitives. Knowing which collective is invoked when is the difference between explaining “DDP all-reduces gradients” and actually understanding the cost of FSDP, TP, or pipeline parallelism. Communication time is often the bottleneck. Collective choice determines throughput.

The five primitives

Let be the number of ranks and be the size of the full logical tensor. A useful communication model is:

where is fixed latency per step. The exact algorithm depends on message size and topology.

Broadcast

One process sends its buffer to all others.

  • Traffic: each receiver obtains bytes. A tree lowers the number of sequential hops for small messages.
  • Use: distribute initial weights, broadcast a hyperparameter.

Reduce

All processes contribute; one receives the sum (or max, min, etc.).

  • Traffic: the root receives a reduced tensor of size . Ring and tree algorithms divide the work differently.
  • Use: aggregating metrics to rank 0 for logging.

All-reduce

All processes contribute and all receive the sum.

  • Ring traffic per rank: , split across reduce-scatter and all-gather phases.
  • Use: gradient aggregation in DDP; output sum in tensor parallelism.

All-gather

Each process has a chunk of size ; all end with the full concatenation.

  • Ring traffic per rank: .
  • Use: FSDP. Gather sharded parameters before forward pass.

Reduce-scatter

Each process contributes a buffer of size ; all end with their slice of the sum.

  • Ring traffic per rank: .
  • Use: FSDP. Sum gradients and keep only your shard.

Identity: all-reduce reduce-scatter all-gather.

The ring all-reduce

The dominant implementation in 2026 (Baidu Ring, Horovod, NCCL):

  1. Each process splits its buffer into chunks.
  2. Reduce-scatter phase ( steps): each process sends one chunk to its right neighbor, receives one from the left, accumulates.
  3. All-gather phase ( steps): each process has the final value of one chunk; cycle around so everyone has the full buffer.

Total: steps, each transferring bytes per rank. Traffic per rank is , which approaches . A ring uses link bandwidth well for large messages. Its many sequential steps can make small messages latency-bound.

One chunk, four ranks

Reduction moves the partial sum; gathering moves the finished chunk.

One chunk traced through a four-rank ring all-reduce In reduce-scatter, rank zero sends its contribution a zero to rank one, which adds a one. Rank one sends that partial sum to rank two, which adds a two. Rank two sends the next partial sum to rank three, which adds a three and finishes chunk A. In all-gather, rank three sends the finished chunk to rank zero, then rank zero forwards it to rank one, and rank one forwards it to rank two without further addition. Every rank concurrently sends one chunk of size B divided by four in each of the six steps, for three B divided by two bytes per rank. REDUCE-SCATTER · 3 hops · receive, then add the local contribution START AT RANK 0 a₀ step 1 AT RANK 1 a₀ + a₁ step 2 AT RANK 2 a₀ + a₁ + a₂ step 3 FINAL A* AT RANK 3 a₀ + a₁ + a₂ + a₃ All four chunks follow the same schedule concurrently; A is isolated here so its accumulation stays visible. ALL-GATHER · 3 hops · forward the finished A* without adding RANK 3 SENDS A* step 1 RANK 0 RECEIVES A* step 2 RANK 1 RECEIVES A* step 3 RANK 2 RECEIVES A* PER-RANK TRAFFIC FOR N = 4 [3 reduce-scatter sends + 3 all-gather sends] × B/4 = 6B/4 = 3B/2
Read it this way: follow chunk A from left to right. Three reduce-scatter hops build A* from four rank-local contributions; three all-gather hops copy A* to the other ranks without more arithmetic. Every rank does this concurrently for one B/4 chunk per step, so it sends 6 × B/4 = 3B/2 bytes, exactly 2B(N − 1)/N for N = 4. Original schematic based on the Baidu ring all-reduce explanation and NCCL collective semantics.

Where each appears

Distributed patternCollectives
DDP (data parallel)All-reduce on gradients per backward pass
FSDP / ZeRO-3All-gather on parameters before forward; reduce-scatter on gradients after backward
Tensor parallelismAll-reduce on activations after each parallel matmul
Pipeline parallelismPoint-to-point sends (not collective) between adjacent stages
Expert parallelism (MoE)All-to-all to route tokens to experts
Embedding lookup at scaleAll-to-all to gather sharded embedding rows

All-to-all

A sixth primitive: each process sends a different chunk to every other process. It is used in MoE token routing and to move a tensor split from one axis to another. Its cost depends on bytes per peer, network bisection bandwidth, routing balance, and topology. It is often a bottleneck, but it is not always more expensive than every other collective.

Hardware backends

  • NCCL (Nvidia): the dominant backend on Nvidia GPUs; ring + tree implementations, NVLink and InfiniBand aware.
  • RCCL: AMD equivalent.
  • MPI: classical HPC backend; used outside ML.
  • Gloo: PyTorch CPU collective backend (slow).

Bandwidth and topology

At least two physical bandwidth levels matter:

  • Inside a fast accelerator domain: links such as NVLink, NVSwitch, or an accelerator torus.
  • Across nodes or slices: networks such as InfiniBand or Ethernet with RDMA.

A large cluster usually has faster local groups and slower links between groups. Hierarchical collectives reduce or gather inside each local group before using the scale-out network. Use measured bandwidth for the target message size rather than a peak link specification.

Cost model in DDP

For a model with parameters, gradient dtype size , and data-parallel ranks, a ring gradient all-reduce has:

  • Logical gradient tensor: bytes.
  • Traffic per rank: bytes.
  • Large-message lower bound: traffic divided by effective ring bandwidth.

For a 7B parameter model with BF16 gradients, GB. On 32 ranks at 200 GB/s effective ring bandwidth, the bandwidth lower bound is about 136 ms. Fixed latency and contention can increase it. Gradient bucketing avoids tiny messages. Overlap with backward can hide part of the reduction, so the exposed time in a step trace is more important than total collective duration.

Common pitfalls

  • All-reducing every parameter separately. Tiny messages are latency-dominated. Use gradient buckets and tune their size, such as PyTorch’s bucket_cap_mb setting.
  • No overlap with compute. PyTorch DDP overlaps automatically; FSDP needs explicit configuration (forward_prefetch, backward_prefetch).
  • Mixed dtypes across ranks. All-reduce requires identical dtype on all ranks; mismatch → cryptic NCCL error.
  • Hangs from rank desync. If one rank skips a collective (e.g., divergent code path), all others hang waiting. Use the same control flow on every rank.
  • Ranking collectives by name alone. Cost follows bytes, message size, algorithm, balance, and topology. All-to-all is difficult on oversubscribed networks, while a very large all-reduce can still cost more.

Found an error or stale claim? Report a correction or send a private note.