Book IV · Chapter 3
Inference and decoding
Move from decoding policy to cache layout, batching, speculation, and test-time compute.
Read first: Transformer architecture and attention
Chapter contents
9 entries · read in order- 01 Decoding strategies: greedy, beam, top-k, top-p, temperature✓ Concept
- 02 KV cache: how LLM inference avoids quadratic decode cost✓ Concept
- 03 Prefill vs. decode: the two phases of LLM inference✓ Concept
- 04 PagedAttention and the vLLM serving model✓ Concept
- 05 Continuous batching for LLM serving✓ Concept
- 06 Speculative decoding✓ Concept
- 07 Quantization: INT8, INT4, FP8, and the inference cost picture✓ Concept
- 08 Test-time compute, search, and verifiers✓ Concept
- 09 Walk me through speculative decoding✓ Question