Fault-tolerant training must restore a coherent optimizer transition, not merely complete a retried collective.
Summary
Do all ranks agree on the operation, input state and whether its optimizer update already happened? Consider two synthetic ranks starting with weight . The intended SGD update subtracts using a fixed gradient.
| Boundary | Rank zero | Rank one |
|---|---|---|
| Committed start | 1 | 1 |
| Partial application, then failure | 0.8 | 1 |
| Blindly apply again | 0.6 | 0.8 |
| Coordinated restore | 1 | 1 |
| Replay from restored inputs | 0.8 | 0.8 |
Replaying communication cannot undo an update already applied on one rank. Even equal weights are insufficient for Adam if moments or step counters disagree.
Failure model first
| Failure | Observable symptom | Core risk |
|---|---|---|
| Fail-stop rank | timeout, connection close, process exit | collective cannot complete |
| Straggler | long tail in operation duration | global step stalls |
| Rank desynchronization | mismatched sequence number or hang | ranks execute different collectives |
| Data corruption | checksum or range mismatch | wrong aggregate may look valid |
| Network partition | subgroups remain alive | split progress or indefinite wait |
| Byzantine participant | arbitrary or adversarial value | aggregate integrity fails |
Fail-stop detection does not provide Byzantine tolerance against arbitrary participant values. A timeout establishes missing timely completion, not the failure’s cause.
Recovery and exactly-once language
Abort the old group and restore all participants from one verified, committed checkpoint manifest. Restore parameters, optimizer, scheduler, scaler, random-generator state, input position and step identity before rebuilding supported membership. Record checkpoint lineage and replayed inputs. The diagram’s once-only result means one coherent intended transition under that protocol, not universal exactly-once execution.
Learning objective
Restore every rank to the committed checkpoint before retrying
Derivation
t happened. The left path repeats work from mixed states; the right path restores one complete boundary first. Replay is coherent only under the stated state-restoration and membership protocol.Can rank zero keep v5 while rank one restores v4 for the retry?
Change the recovery contract
Continue with fewer ranks
New task: resume after losing one data-parallel worker.
Picture and change: equal-sized local gradients previously averaged to six. The surviving gradients two and four average to three, not six.
Check: dividing their sum by the old world size gives two, which is not even the surviving-batch mean. Recompute normalization and effective batch; restore compatible state and record the changed experiment. A missing required tensor shard needs reconstruction, not fewer-gradient averaging.
Use when: the runtime supports membership rebuild and the model’s decomposition remains feasible.
Admit an arbitrary participant
New task: a live rank can submit a finite but wrong gradient.
Picture and change: gradients two and one hundred average to fifty-one. Both can pass shape, finiteness and byte-integrity checks.
Check: adding another honest value two makes the median two, but changes the aggregation rule. That rule needs an explicit honest-participant and statistical model; heterogeneous legitimate gradients can also be suppressed.
Use when: the threat exceeds fail-stop transport detection. Separate integrity checks, redundant computation and statistically robust aggregation.
Reconstruct the boundary
Can matching checksums establish that the restored gradient is correct?
Reasoned answer
No. Matching bytes can encode the same wrong computation. Check agreement and checkpoint completeness separately from numerical correctness and experiment equivalence. Equal weights with unequal optimizer moments can still produce different next updates.
Recovery choices
Abort and restart
Restore a committed consistency domain, not each rank’s newest independently written file. An incomplete or corrupted checkpoint can preserve the original failure.
Elastic membership
Check data partition, loss denominator, optimizer state redistribution and tensor/pipeline group constraints. Availability does not imply the original trajectory is preserved.
Redundancy
Replicas, coded computation and redundant shards consume resources and require a specified reconstruction rule. Common-mode faults can defeat supposedly independent copies.
Approximate aggregation
Partial or asynchronous progress changes which gradients enter an update and their staleness. Treat convergence and bias as new obligations, not free availability.
Detection
Capture process-group identity, operation sequence, shape/dtype, rank starts/ends, deadlines, hardware errors and liveness. Retain parameter/optimizer/step checks and checkpoint-manifest completeness. Preserve evidence before teardown. Transport retries cannot restore application state.
Silent corruption
Checksums detect specified byte changes; redundant calculation tests different faults. Neither guarantees arbitrary computational correctness. Trimmed means and coordinate medians need stated assumptions and can distort legitimate stochastic gradients.
Common confusions
Independent zero-mean noise may average down; correlated or adversarial errors need not. Do not confuse transport completion with global optimizer publication. Framework/version-specific group reconstruction is not a portable blanket retry promise.
In an interview
State the failure model, committed boundary and recovery consistency domain. Then specify detection, validation, supported membership and lineage before claiming the experiment survived unchanged.
Related: all-reduce and collectives, FSDP and ZeRO, and design fault-tolerant distributed training.