Skip to content
mentorship

Inference systems work sample

LLM inference scheduler lab

The hard part is not naming vLLM. It is turning latency, throughput, fairness, and KV capacity into one explicit scheduling policy.

75 minutes plus 30-minute design follow-upScheduler starter, tests, and an arrival trace

Do this only if the format matches

Confirm the round with recruiting first. This is an original practice exercise, not a claim that a specific company uses this exact prompt.

Protocol

  1. Derive worst-case KV reservation per request before writing scheduler code.
  2. Implement admission and release invariants.
  3. Keep decode moving while chunking long prefills.
  4. Rotate across tenants without violating per-tenant FIFO.
  5. Run the supplied tests, then replay the arrival trace conceptually.
  6. Defend overload shedding, SLO metrics, and one alternative scheduler.

Deliverables

  • A passing scheduler implementation.
  • A policy note for time to first token and inter-token latency.
  • An overload and tenant-isolation design.
  • A bottleneck dashboard with four discriminating metrics.

Graduation gates

01

Reserved blocks never exceed capacity.

02

Long prefill cannot starve active decode.

03

Finished requests release memory immediately.

04

The policy has an explicit rejection or degradation point.

Starter files

Assignment READMESystem contract and follow-ups.Open file →Scheduler starterAdmission and iteration scheduling.Open file →Public testsCapacity, fairness, and chunking.Open file →Arrival traceMixed prompts, tenants, and generations.Open file →

Answer key is a rubric, not a solution dump

The linked question explains what strong performance looks like, common failure modes, and senior follow-ups. Attempt the lab before reading it.

Read the full system-design answer →

← All executable labs