Skip to content
mentorship

LLM training debugging work sample

Broken frontier LLM training lab

A training run that executes is not a training run that is correct. Repair semantics before tuning optimization.

75 minutesPyTorch package, failing contract tests, source review, one-batch overfit

Do this only if the format matches

Confirm the round with recruiting first. This is an original practice exercise, not a claim that a specific company uses this exact prompt.

Protocol

  1. Run contract tests and classify failures as data, model, update, or evaluation defects.
  2. Trace one token through input, target, mask, logits, loss, and gradient.
  3. Repair label and causal-mask semantics before changing the optimizer loop.
  4. Make accumulation, clipping, and scheduler cadence agree on what an update means.
  5. Restore evaluation isolation and caller mode.
  6. Overfit one tiny batch, then record the first remaining multi-GPU risk.

Deliverables

  • Repaired code and one new regression test.
  • A defect table with symptom, mechanism, and verification.
  • A one-batch overfit result.
  • A prioritized plan for a distributed recurrence.

Graduation gates

01

Targets are next tokens and padded targets are ignored.

02

Prefix outputs cannot depend on future tokens.

03

Cross-entropy receives raw logits.

04

One accumulation window creates one optimizer and scheduler step.

Starter files

Answer key is a rubric, not a solution dump

The linked question explains what strong performance looks like, common failure modes, and senior follow-ups. Attempt the lab before reading it.

Read the diagnostic framework →

← All executable labs