Book V · Chapter 2
Distributed training and parallelism
Build from collectives and sharded matrix operations to multi-axis parallel plans.
Read first: Hardware and performance
Chapter contents
7 entries · read in order- 01 All-reduce and other collectives✓ Concept
- 02 Sharded matrix multiplication✓ Concept
- 03 FSDP and ZeRO: sharding optimizer state, gradients, and parameters✓ Concept
- 04 Tensor parallelism✓ Concept
- 05 Pipeline parallelism✓ Concept
- 06 Context parallelism and ring attention✓ Concept
- 07 Strong scaling, MFU, and parallelism selection✓ Concept