Book II
Model training and research
Optimization, reliable experiments, implementation, debugging, and research judgment.
9 chapters · 52 entriesChapters
9Objectives and regularization
Start with training objectives, then control fit and generalization.
Optimization and schedules
Move from SGD and Adam to schedules, clipping, and initialization.
Numerics and training stability
Understand normalization, precision, floating-point behavior, and loss failures.
Efficiency, data, and scaling
Reduce memory and work, build training data, and reason about scale.
Training recipe and case study
Connect the parts into one training plan, then inspect a real open training log.
Training and debugging questions
Practice diagnosis, numerical decisions, imbalance, sequence training, and optimizer trade-offs.
ML implementation
Implement core algorithms and transformer mechanisms with tests and explicit contracts.
Training code and codebase debugging
Debug a training loop and extend an unfamiliar ML repository safely.
Research methods and derivations
Derive, critique, design ablations, and investigate behavior from evidence.