Skip to content
mlmentorship

GPU memory hierarchy: HBM, SRAM, and roofline reasoning

Decide whether an accelerator operation is limited by compute or by data movement across HBM, caches, and on-chip memory.

Published · 5 min read ·Role-specific ·Advanced

Visual quick review

Visual first · depth when needed

Use arithmetic intensity to explain why batching raises decode's bandwidth-limited throughput ceiling and why only operations beyond the hardware ridge become compute-bound.

Preparing the visual…

Summary

A GPU has small, fast on-chip memory and larger, slower HBM. Large matrix multiplications are often compute-bound. Decode, small matrix multiplications, and many elementwise operations are often memory-bound. The operation’s arithmetic intensity determines which limit applies.

Counting only multiply-adds can give the wrong answer. A better first-order model is:

For most LLM ops at typical batch sizes, the second term dominates. This single fact explains:

  • Why FlashAttention is faster despite doing the same FLOPs (reduces HBM traffic).
  • Why decoding is slow with batch 1 even though the model “fits” (memory-bound).
  • Why batching helps decoding so much (amortizes weight reads across many tokens).
  • Why low-precision data types help even when the matmul is already fast (less HBM bandwidth).

The hierarchy

TierCapacityBandwidthLatencyWhat lives here
Register filehundreds of KB per SMhighest on-chipvery lowper-thread values
SRAM (shared memory / L1)tens to hundreds of KB per SMvery highlowtiles for matrix operations
L2 cachetens to hundreds of MB per devicebelow local SRAMmediumshared across SMs
HBMtens to hundreds of GBbelow on-chip memoryhigherweights, activations, KV cache
PCIe or accelerator linkexternalbelow local HBM in many systemsµs scalehost or peer transfer

These values are illustrative. Capacity, bandwidth, precision, sparsity mode, and power settings vary by accelerator and SKU. Use measured values from the target machine for a deployment plan.

Arithmetic intensity

For a kernel doing FLOPs and moving bytes between HBM and SRAM:

A kernel is compute-bound when its intensity exceeds the GPU’s peak FLOPs/byte ratio (the “ridge” in a roofline plot). Otherwise it’s memory-bound.

An accelerator with 989 BF16 TFLOPs/s and 3 TB/s of HBM bandwidth has a ridge near 330 FLOPs/byte. For comparison:

Learning objective

When does more compute stop waiting on HBM?

Illustrative GPU roofline with decode, batching, a ridge point, and a large matrix multiplication A log-scale roofline for an illustrative accelerator with 989 BF16 teraFLOPs per second and 3 terabytes per second of HBM bandwidth. The sloped bandwidth ceiling reaches the flat compute ceiling at 330 FLOPs per byte. Batch-one decode is near one FLOP per byte. Reusing each weight across a batch of 64 moves decode right to roughly 64 FLOPs per byte but leaves it below the ridge. A 2048 by 2048 square matrix multiplication can reach roughly 1024 FLOPs per byte and lies to the right of the ridge. HBM roof = BW × intensity compute ceiling = 989 TFLOP/s MEMORY-BOUND COMPUTE-BOUND decode · batch 1 ≈ 1 FLOP/byte → ceiling ≈ 3 TFLOP/s batching reuses each weight batch 64 ≈ 64 FLOPs/byte ridge ≈ 330 2048² matmul · AI ≈ 1024 1643301000 arithmetic intensity (FLOPs per HBM byte, log scale) attainable throughput (log scale) Upper bounds only; profile the target kernel.
Read it this way: start at batch-1 decode: one weight byte supports little work, so the sloped HBM roof is far below peak compute. Batching reuses the loaded weights and moves the operation right, raising its bandwidth-limited ceiling. Only after arithmetic intensity crosses roughly 330 FLOPs/byte does the flat 989-TFLOP/s compute roof become the first-order limit. The log-scale construction and worked points are original; the model is checked against the original Roofline report, NERSC's Roofline documentation, and NVIDIA's matrix-multiplication guide.
  • Large square matmul ( by ): up to about FLOPs/byte when counting two BF16 inputs and ignoring output traffic. Counting output reads and writes lowers the intensity.
  • Attention kernel (without FlashAttention): ~O(d) FLOPs/byte → memory-bound at common d=64–128.
  • Single-token decode with BF16 weights: about 1 FLOP per weight byte at batch 1, before other traffic. One multiply-add uses a two-byte weight. This is severely memory-bound.

Implications for LLMs

  • Training: large matrix multiplications can be compute-bound, while small batches, elementwise work, and communication can expose other limits.
  • Decode: often weight-bandwidth-bound at small batch sizes. Batching amortizes weight reads until matrix compute or KV-cache traffic becomes the next limit.
  • KV cache: bandwidth-dominated read at every decode step. Cache size growth is a serving-throughput issue.

Common pitfalls

  • Quoting FLOPs as a single-number cost. Throughput on memory-bound kernels is dictated by bytes, not FLOPs.
  • Assuming peak compute predicts every speedup. A new accelerator can increase matrix throughput faster than memory bandwidth. Memory-bound work then sees a smaller gain.
  • Ignoring on-chip memory when designing kernels. FlashAttention tile sizes are constrained by shared memory, registers, and the compiled kernel, not only HBM size.

Found an error or stale claim? Report a correction or send a private note.