Summary
Quantization stores or computes selected tensors with a reduced code set and an explicit reconstruction policy. Rounding, clipping, scale granularity, metadata, and supported kernels determine numerical and systems effects separately. Smaller weights do not guarantee faster execution or a universal bound on task-quality change.
The mechanism
For positive scale and integer zero-point , affine quantization uses:
The pictured symmetric INT4 fixture uses , codes through , and half ties rounded away from zero. It deliberately excludes code ; other four-bit schemes need not use this convention. With shared , ordinary values become codes . Reconstruction is , with squared error .
Giving that channel yields codes and reconstruction . Its squared error becomes . The high-range row keeps under either policy, restoring .
Learning objective
Separate low-bit fixture: shared versus channelwise scale granularity
An original symmetric INT4 example compares one scale shared by two channels with a separate scale for each channel. Values are rounded to integer codes from negative seven through seven and then dequantized.
One shared scale
The high-range channel sets Δ = 8/7 ≈ 1.14 for both rows.
| Channel | Original | Codes → restored |
|---|---|---|
| ordinary | −1, −0.5, 0.5, 1 | −1, 0, 0, 1 → −1.14, 0, 0, 1.14 |
| high range | −8, −4, 4, 8 | −7, −4, 4, 7 → −8, −4.57, 4.57, 8 |
Lost detail: −0.5 and 0.5 both collapse to code 0.
One scale per channel
The ordinary row gets Δ = 1/7; the high-range row keeps Δ = 8/7.
| Channel | Original | Codes → restored |
|---|---|---|
| ordinary | −1, −0.5, 0.5, 1 | −7, −4, 4, 7 → −1, −0.57, 0.57, 1 |
| high range | −8, −4, 4, 8 | −7, −4, 4, 7 → −8, −4.57, 4.57, 8 |
Recovered detail: each row uses the full signed code range.
Reference drawing
Learning objective
One outlier can erase small values under a shared quantization scale
Derivation
An original symmetric INT4 example compares one scale shared by two channels with a separate scale for each channel. Values are rounded to integer codes from negative seven through seven and then dequantized.
One shared scale
The high-range channel sets Δ = 8/7 ≈ 1.14 for both rows.
| Channel | Original | Codes → restored |
|---|---|---|
| ordinary | −1, −0.5, 0.5, 1 | −1, 0, 0, 1 → −1.14, 0, 0, 1.14 |
| high range | −8, −4, 4, 8 | −7, −4, 4, 7 → −8, −4.57, 4.57, 8 |
Lost detail: −0.5 and 0.5 both collapse to code 0.
One scale per channel
The ordinary row gets Δ = 1/7; the high-range row keeps Δ = 8/7.
| Channel | Original | Codes → restored |
|---|---|---|
| ordinary | −1, −0.5, 0.5, 1 | −7, −4, 4, 7 → −1, −0.57, 0.57, 1 |
| high range | −8, −4, 4, 8 | −7, −4, 4, 7 → −8, −4.57, 4.57, 8 |
Recovered detail: each row uses the full signed code range.
Calibrate for a Larger Outlier
New task: Deployment introduces value sixteen in the high-range channel.
Picture and change: Retain the old scale and clip, or recalibrate to and recompute all affected codes.
Check: The stale scale clips sixteen to code seven and restores eight, absolute error eight. With a shared recalibrated scale, the entire ordinary row rounds to zero and its squared error becomes .
Use when: Deployment ranges exceed calibration coverage; retaining separate scales can isolate this particular cross-channel effect.
Budget Group-Scale Overhead
New task: Store 128 four-bit weights with one BF16 scale.
Picture and change: Pack sixty-four payload bytes and retain two scale bytes; the kernel must use the same grouping.
Check: Sixty-six bytes versus 256 BF16 bytes is times smaller, excluding headers, alignment, and zero-point metadata.
Use when: Granularity improves reconstruction while memory remains constrained; benchmark actual dequantization and task quality separately.
The flavors
Weight-only quantization (most common)
Compress weights while declaring activation, compute, and accumulation dtypes separately. The retained heading does not establish a deployment-frequency ranking.
Activation quantization (less common, harder)
Declare W8A8’s formats, scales, and accumulators. FP8 E4M3 and E5M2 trade precision against range differently and are not affine integer codes.
KV cache quantization
Compress cached K/V with a supported layout and evaluate long-context behavior, not only short prompts.
Quantization-aware training (QAT)
Model quantization effects during training; compare its extra training cost and quality against suitable post-training baselines.
What an interviewer expects you to say
Name the compressed tensor, rounding/scaling contract, target kernel, and quality evaluation before choosing a method.
Where the wins come from
Profile weight bandwidth, arithmetic, cache capacity, and dequantization; the bottleneck changes with batching, context, hardware, and kernels.
Common confusions
GPTQ uses calibrated second-order reconstruction approximations; AWQ uses activation-aware scaling, not mandatory higher-precision protection. SmoothQuant redistributes scale across activations and weights; GGUF is a storage format supporting multiple schemes, not one quantization algorithm.
Picking a quantization method (decision tree)
Measure supported candidates and sensitive slices; keep higher-precision tensors where justified rather than treating one method as universally best.
Why interviewers ask
Connect byte arithmetic to a measured bottleneck and a task-quality acceptance criterion.
Related: Mixed precision training, KV cache, How to think about LLM inference cost.