GPU memory hierarchy: HBM, SRAM, and roofline reasoning

Summary

A kernel must perform arithmetic and move data. Count both for the same operation before choosing an optimization.

Consider 120 GFLOPs and 12 GB transferred across the measured memory boundary. Assume hypothetical ceilings of 20 TFLOP/s at the operation’s precision and 100 GB/s across that boundary. Use decimal units: one GB is $1 0^{9}$ bytes. Which could help: twice the compute throughput, or half the traffic?

The compute and bandwidth lower bounds are $120/20000 = 0.006$ seconds and $12/100 = 0.12$ seconds. Even perfect overlap cannot finish below the larger bound:

T \geq max (\frac{F}{P}, \frac{B}{W}) .

Here $F$ counts FLOPs, $B$ counts transferred bytes, $P$ is a compute ceiling, and $W$ is a bandwidth ceiling. Keep units consistent; a multiply-add conventionally counts as two FLOPs.

Change	Compute bound	Traffic bound	Combined lower bound
Original	$0.006$ s	$0.12$ s	$0.12$ s
Double $P$	$0.003$ s	$0.12$ s	$0.12$ s
Halve $B$	$0.006$ s	$0.06$ s	$0.06$ s

Halving traffic improves the lower bound; it does not prove a measured twofold speedup.

Arithmetic intensity

Arithmetic intensity is work per transferred byte, at a stated memory boundary:

arithmetic intensity = \frac{F}{B} (FLOPs per byte)

The fixture has intensity $I = 10$ FLOPs/byte and ridge $P / W = 200$ FLOPs/byte. Its bandwidth roof is lower than its compute roof:

e x t t h r o ug h p u t \leq min (P, W I) .

The lower roof identifies a first-order limit, not every possible bottleneck.

Same operation, two resource limits

Halving memory traffic raises this example's ceiling from 1 to 2 TFLOP/s

Derivation

Read it this way: halving bytes doubles arithmetic intensity and its bandwidth ceiling. Both points remain below the compute roof. Doubling compute alone leaves their lower bandwidth roof unchanged.

Check: Why does halving HBM traffic double this ceiling without proving a measured speedup?

Read it this way: At one TB/s, doubling arithmetic intensity doubles the memory ceiling. Other bottlenecks may prevent reaching either ceiling.
Does a doubled memory ceiling prove an observed throughput doubled?

Model references: original Roofline report, NERSC Roofline documentation, and NVIDIA matrix-multiplication guide.

Change reuse or local capacity

1. Reuse weights across a batch

New task: small-batch decode repeatedly reads the same BF16 weights.

Picture and change: one multiply-add per two-byte weight gives about one FLOP per weight byte. Reusing each weight across 64 tokens gives about 64, counting weight traffic alone.

Check: include activations, outputs and each sequence’s KV reads. Batching also consumes memory and can increase waiting time.

Use when: measured weight traffic limits small-batch throughput.

2. Reject a tile that cannot stay on chip

New task: two BF16 input tiles require $2 \times 12 8^{2} \times 2 = 65536$ bytes. The kernel also needs a $12 8^{2}$ FP32 accumulator: another 65536 bytes.

Picture and change: those arrays cannot all reside in a hypothetical 96 KiB local budget. Choose smaller tiles or a different staging plan. Registers and shared memory have separate constraints; this combined check is only a necessary fit test.

Check: count spills, occupancy, synchronization, output traffic and compiled storage use before claiming reuse.

Use when: local capacity invalidates the proposed traffic reduction.

A faster compute ceiling can leave the limit unchanged

Predicting decode speedup from peak tensor throughput alone ignores bytes and launch costs. Record shapes, dtype, FLOP convention, memory boundary, traffic, attainable rates and measured runtime. Compare the same operation and distinguish useful arithmetic from executed or recomputed work.

Try finding the missing limit

Why might measured runtime exceed both simple lower bounds?

Check your answer

Launch overhead, dependencies, low occupancy, contention, inefficient instructions or uncounted traffic can add limits. The roofline is not a complete runtime simulator.

The hierarchy

Tier	Typical role	Constraint to inspect
Registers	Per-thread values and accumulators	Register allocation, spills, occupancy
Shared memory / L1	On-chip staging and cached values	Distinct APIs and architecture-specific partitioning
L2	Cache shared across multiprocessors	Actual hit rate and traffic boundary
HBM	Weights, activations, KV storage	Capacity and achieved bandwidth
PCIe / accelerator links	Host or peer transfers	Link, topology and synchronization

Capacities and rates vary by device, precision, sparsity mode and power settings. SRAM is a storage technology, not one universally managed tier. Measure the target machine and kernel.

Other intensity calculations

For square BF16 matrix multiplication, $F \approx 2 n^{3}$ and two input reads cost $4 n^{2}$ bytes. Ignoring outputs gives $I \approx n /2$ ; output reads, writes or repeated loads lower that estimate. At $n = 2048$ , this input-only estimate is 1024 FLOPs/byte.

The separate illustrative 989 TFLOP/s, 3 TB/s device has ridge about 330 FLOPs/byte. It is not the fixture device or a measured kernel result.

Implications for LLMs

Large matrix multiplications can approach a compute roof; small operations, communication and elementwise kernels can face other limits. Decode can become limited by weights or KV reads, depending on batch, context and implementation. FlashAttention reduces materialized attention traffic through tiling; it can also change executed work through recomputation.

Common pitfalls

Peak figures do not replace profiling. Include output traffic, spills, cache behavior and the specific boundary at which bytes were measured.

Accelerator network topology. Extend the hierarchy across devices and nodes.
Transformer compute and memory accounting. Convert a model configuration into bytes and FLOPs.
Profiling distributed ML workloads. Compare a trace with roofline limits.