Skip to content
mlmentorship

Walk me through how you'd train a 100B parameter model

The question is about parallelism and memory, not about modeling. The L6 answer combines data, tensor, pipeline, and FSDP/ZeRO sharding into a coherent strategy.

Published · 7 min read ·Role-specific ·Advanced

Systems / infrastructure · active recall

Practice before you read

35 minutes. Quantify the workload, identify the bottleneck, choose an architecture, and design failure recovery.

Back to my workbook How practice works

Systems / infrastructure · closed-book attempt

Walk me through how you'd train a 100B parameter model

Quantify the workload, identify the bottleneck, choose an architecture, and design failure recovery.

35:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

30-second answer map

Visual first · depth when needed

Distinguish data, tensor, and pipeline parallelism as composable compute-placement axes from FSDP or ZeRO-3 as the state-sharding policy applied across data-parallel ranks.

Preparing the visual…

Asked in: senior LLM-team and infra-leaning ML loops.

The question tests systems thinking. The L4 answer says “use distributed training.” The L6 answer separates data, tensor, and pipeline parallelism from state-sharding methods, then describes the right combination for the model and hardware.

What an L4 answer sounds like

“I’d use multiple GPUs and split the data across them. Maybe use Horovod or PyTorch DDP.”

This works for a 1B model, not for a 100B model. Pure data parallelism requires the full model to fit on each GPU; a 100B model in BF16 is 200 GB, more than any single GPU. You’ve trained at scale only in the textbook sense.

Placement axes versus stored state

What does each parallel technique divide?

Three compute-placement axes and one state-sharding policy Tensor parallelism divides one layer operation into cooperating tensor slices. Pipeline parallelism divides model depth into sequential stage groups. Data parallelism gives model replicas different mini-batches. FSDP or ZeRO-3 is not a fourth compute axis: across the data-parallel group, each rank persistently stores a parameter, gradient, and optimizer-state shard, gathering the current layer for compute and reduce-scattering its gradients afterward. TP · inside one layer operation activation same input slice 1 slice 2 slice 3 collective within the operation PP · across model depth stage 1 early layers stage 2 middle layers stage 3 late layers DP · across mini-batches replica A full compute · batch A replica B full compute · batch B gradient synchronization joins replicas FSDP / ZeRO-3 · stored state across the DP group DP rank A state shard 1 of 2 DP rank B state shard 2 of 2 all-gather reduce-scatter storage + communication policy — not a fourth axis
Read it this way: first choose who computes each tensor slice (TP), layer range (PP), and mini-batch (DP). Then choose what state persists on the DP ranks: FSDP or ZeRO-3 shards it and temporarily gathers the current layer. These choices compose; FSDP is not a fourth mesh dimension. Original schematic, checked against the Megatron-LM and ZeRO papers.

What an L5 answer sounds like

“100B parameters in BF16 is 200 GB of weights, plus gradients (200 GB), plus two Adam moments in FP32 (800 GB), plus activations. Some stacks also keep 400 GB of FP32 master weights. None of this fits on a single GPU. The training stack may need several placement and state-sharding methods:

  1. Data parallelism (DP): replicate the model across GPUs, split the batch. Each GPU gets a different mini-batch slice. Gradients are all-reduced.
  2. Tensor parallelism (TP): split individual matrix multiplications across GPUs. Useful for the large MLPs and attention layers. Requires fast interconnect (NVLink, not just PCIe).
  3. Pipeline parallelism (PP): split the model by layer across GPUs. Each stage processes micro-batches in pipelined fashion. Has bubble overhead.
  4. Sharded data parallelism (FSDP / ZeRO Stage 3): change the data-parallel axis so it also shards model weights, gradients, and optimizer state. This is a data-parallel storage method, not an independent mesh axis.

A reasonable starting recipe for 100B is TP on the fastest links, optimizer or full-state sharding across data-parallel ranks, and PP only if memory, depth, or topology requires it. Add BF16, activation checkpointing, and gradient accumulation to reach the intended effective batch size. Then measure a small layout grid.”

This is L5. You’ve named the parallelism axes, sized the memory problem, and given a defensible recipe.

What an L6 answer sounds like

“…a few more practical considerations:

Hardware topology drives the strategy. TP communicates activations every layer, so place it on the fastest useful links. PP communicates only between adjacent stages. FSDP moves parameters and gradients and often relies on overlap with compute. Use measured collective bandwidth for the target message sizes rather than peak link specifications.

Optimizer state is a major memory cost. Adam’s two FP32 moments use four times the bytes of BF16 weights. ZeRO Stage 1 shards optimizer state. Stage 2 adds gradient sharding. Stage 3 adds parameter sharding. Use the lightest stage that leaves enough room for activations and buffers.

Activation memory at long context dominates everything. For long-sequence training, activations exceed weights. Mitigations: activation checkpointing (recompute during backward), sequence parallelism (split activations along the sequence axis across TP ranks), context parallelism (Ring Attention, etc.).

Throughput must be measured. Communication overlap, kernel shapes, and micro-batch size can change throughput on the same hardware. Record MFU, exposed collective time, bubbles, peak memory, and per-rank imbalance.

Stability is the real challenge at this scale. Loss spikes are common; the standard mitigations are gradient clipping, careful warmup, sometimes embedding norm or QK norm. Numerical issues that don’t appear at 1B can be fatal at 100B.”

Tells that get you a strong-hire vote

  • You compute the memory budget explicitly (weights, gradients, optimizer state, activations).
  • You explain data, tensor, and pipeline parallelism, plus the state-sharding choice within data parallelism.
  • You map them to hardware topology based on communication frequency and measured links.
  • You mention activation memory as a separate concern at long context.
  • You discuss stability (loss spikes, gradient clipping) as a first-order concern at scale.

Tells that get you down-leveled

  • “Just use DDP” (won’t fit).
  • Confusing TP, PP, FSDP.
  • No memory budget calculation.
  • Suggesting CPU offloading as the primary strategy (works, but slow; usually a last resort).
  • No mention of activation checkpointing.

Common follow-up

“What’s the difference between FSDP and tensor parallelism?”

The L6 answer:

“TP shards an individual matrix multiplication across GPUs. The matrix is logically one operation; physically, each GPU computes part. Communication happens during the operation (all-reduce after each TP layer).

FSDP shards the weights across data-parallel ranks. Each GPU stores a partition; before a layer’s forward pass, the full weight is gathered from the partitions. Communication happens between layers (all-gather before each layer, reduce-scatter for gradients).

They’re complementary: TP gives per-operation parallelism with frequent activation communication. FSDP gives persistent per-layer memory savings through parameter gathers and gradient reduce-scatter. Real systems can combine both when the memory and topology estimates support it.”

Related: Transformer compute and memory accounting, strong scaling and parallelism selection, and plan a 70B training run.

Found an error or stale claim? Report a correction or send a private note.