Pruning: structured vs unstructured sparsity

Summary

Pruning removes weights from a trained network and fine-tunes to recover accuracy. Unstructured pruning zeros individual weights. Structured pruning removes entire neurons, channels, heads, or layers.

Modern networks are massively over-parameterized. The lottery ticket hypothesis (Frankle & Carbin, 2019) suggests that within a trained dense network, a small subnetwork (10 to 20 percent of weights) reaches the same accuracy when retrained. Pruning is the practical exploitation of that observation.

Two distinct goals:

Smaller model on disk and in memory: unstructured pruning + sparse storage. Useful for distribution and memory-bound deployment.
Faster inference on real hardware: structured pruning. Removes whole tensors so the remaining computation is dense and matches GEMM kernels.

Learning objective

Zero weights leave dense shapes unchanged; removing channels shrinks them

Derivation

Unstructured: 8 weights, logical shape 4 x 4

Individual connections become zero, but every input-output position remains in the operator.

Input	Out 1	Out 2	Out 3	Out 4
In 1	w	0	w	0
In 2	0	w	0	w
In 3	w	w	0	0
In 4	0	0	w	w

Dense path still sees 4 x 4; sparse speed needs a compatible sparse kernel.

Structured: 8 weights, executed shape 4 x 2

Remove output channels 2 and 4, then pack the survivors into a smaller dense matrix.

Input	Out 1	Out 3
In 1	w	w
In 2	w	w
In 3	w	w
In 4	w	w

Standard dense path now executes 4 x 2 and produces two output channels.

both examples retain eight weights, but only structured pruning changes this dense operator from 4 x 4 to 4 x 2. A zero count saves compute only when storage and hardware kernels can exploit its pattern.

Check: Why does the dense path shrink only in the structured example? What output-shape change must downstream layers accommodate?

Read it this way: Unstructured zeros still occupy this dense 3x3 matrix. Removing an entire channel changes the shape and can remove a whole row.
Does filling a channel with zeros change the dense matrix shape?

The two regimes

Unstructured pruning

Zero individual weights below some magnitude threshold. Common criterion: magnitude pruning ( $∣ w ∣ < τ$ ), often combined with weight decay during fine-tuning.


Sparsity achievable	90 to 95 percent on overparameterized models
Storage benefit	Real (CSR / CSC formats)
Speed benefit	None on standard GPUs
Why no speedup	Sparse matmul kernels are rarely faster than dense matmul until > 90 percent sparsity, and only on specialized hardware (NVIDIA 2:4 sparsity, custom accelerators)

Structured pruning

Remove entire structures: convolutional channels, transformer heads, FFN neurons, sometimes whole layers.


Sparsity achievable	30 to 70 percent typical
Storage benefit	Real
Speed benefit	Real, proportional to sparsity
Why it works on hardware	Output is a smaller dense tensor; runs through standard GEMM

The 2:4 semi-structured compromise

NVIDIA Ampere and later support 2:4 sparsity: in every group of 4 weights, at least 2 are zero. Sparse Tensor Cores can provide up to 2x the math throughput of the equivalent dense operation; end-to-end speedup is lower and workload-dependent. This is a compromise between unstructured pruning (more flexible, but requiring general sparse kernels) and coarse structured pruning (less flexible, but producing smaller dense tensors).

Pipeline

Train the dense model normally.
Score weights or structures by importance (magnitude, gradient-magnitude, Hessian-based, Fisher).
Prune below a target sparsity.
Fine-tune to recover accuracy. Often iterative: prune-finetune-prune-finetune.
Optional: lottery-ticket rewind. Reset weights to an early-training checkpoint, train the sparse mask from there.

Tradeoffs vs other compression

Pruning vs quantization: orthogonal. Combine both. INT8 + 50 percent structured sparsity is common in production.
Pruning vs distillation: distillation trains a smaller model from scratch with a teacher’s soft targets. Pruning starts dense and shrinks. Distillation often produces better small models but needs the teacher’s training data.
Pruning at training time (e.g. RigL, Sparse Transfer): grow-and-prune during training. Avoids the prune-finetune cycle but harder to tune.

Common pitfalls

Reporting unstructured-sparsity speedups on standard GPUs. Almost always misleading. Sparse storage is not sparse compute.
Pruning before training is done. Pruning the trajectory of training, not the final model, often hurts.
Treating “90 percent sparse” as a quality measure. What matters is task performance at the achieved compute or memory cost.
Forgetting batchnorm / layernorm. Structured pruning needs to also adjust normalization statistics for the kept channels.