The GPU Memory Hierarchy in Numbers
An H100 SXM has 33 MiB of registers, up to 29 MiB of shared memory, a 50 MB L2 and 80 GB of HBM3 at 3.35 TB/s. Knowing those capacities decides which inference tensors can be reused on chip, why decode streams every weight from HBM each step, and when a benchmark is secretly measuring L2.
TL;DR: On an H100 SXM the on-chip stores are tiny next to the model: 33 MiB of registers, at most 29.4 MiB of shared memory and a 50 MB L2, against 80 GB of HBM3 at 3.35 TB/s. One Llama 3.1 8B layer is 436 MB in BF16, nearly nine times the L2, so every decode step streams all 16 GB of weights from HBM and takes at least 4.8 ms. Kernel design is the art of reusing bytes at the level closest to the tensor cores.
The levels
Four levels matter for inference kernels. Figures are for the H100 SXM from NVIDIA's Hopper architecture documentation and datasheet: 132 SMs, each with a 256 KB register file (65,536 32-bit registers) and 256 KB of combined L1 and shared memory, of which up to 228 KB can be configured as shared memory.
| Level | Per SM | Whole GPU | Who manages it | What inference keeps there |
|---|---|---|---|---|
| Registers | 256 KB | 33 MiB | compiler | matmul accumulators, softmax state |
| Shared memory | up to 228 KB | 29.4 MiB | the kernel | Q, K, V tiles; GEMM operand stages |
| L2 | shared | 50 MB | hardware | recently touched lines; small matrices |
| HBM3 | shared | 80 GB | the engine | weights, KV cache, activations |
Two facts in that table surprise people. The register file is the largest on-chip store, slightly larger in total than shared memory, which is why FlashAttention keeps its output accumulator in registers rather than in shared memory. And the gap from L2 to HBM is a factor of 1,600 in capacity, so for an LLM the L2 is a staging buffer, never a cache for the model.
Per-level bandwidths and latencies are not on the datasheet. What is published is HBM: 3.35 TB/s across 132 SMs is about 25 GB/s per SM. Every on-chip level is faster than that, which is the entire reason to tile.
Worked example: where a decode step's bytes live
model Llama 3.1 8B, BF16, batch 1, H100 SXM
weights 8,030,261,248 params x 2 B = 16.06 GB
one layer q 4096x4096 + k,v 2 x 4096x1024 + o 4096x4096
+ MLP 3 x 4096 x 14336 = 218.1 M params
x 2 B = 436 MB
vs L2 50 MB / 436 MB = 11% of one layer
decode each weight used once per step (batch 1), no reuse to exploit
floor 16.06e9 B / 3.35e12 B/s = 4.79 ms per token
sanity: 1 / 4.79 ms = 209 tokens/s at 100% of HBM bandwidth; real MBU is lower
The same capacities decide what fits on chip in a compute-bound kernel. A FlashAttention tile of 128 queries by head dim 128 in BF16 is 32 KiB; a K tile and a V tile are another 32 KiB each, so Q plus two pipelined stages of K and V is 160 KiB, inside the 228 KB shared memory budget. The FP32 output accumulator for that tile is 64 KiB, which is why it lives spread across the registers of the warps that own the rows.
Reuse is the only lever
The roofline ridge on this part is 989 / 3.35 = 295 FLOP/byte (dense BF16). A kernel gets above it only by loading a byte from HBM once and using it many times from a faster level. A GEMM at large batch does that: each weight tile in shared memory feeds every row of the activation tile. A batch-1 GEMV cannot, since each weight is used for one multiply-add, so its intensity is 1 FLOP per byte and it lives or dies by HBM bandwidth. Attention in prefill reuses each K and V tile across a block of queries; attention in decode reuses each KV tile only across the query heads of one GQA group, four for Llama 3.1 8B.
Where it breaks
The figures above are H100 SXM. Other parts change the capacities and the SM count, so redo the arithmetic per GPU; a B200 has 180 GB of HBM at 8 TB/s and a different on-chip budget. The L2 is split into two partitions on Hopper, and an SM reading the far partition pays extra latency. The capacity of shared memory per SM is shared by every resident thread block, so a kernel that asks for 200 KB of shared memory runs one block per SM and loses the latency hiding that several resident blocks would give. Register use per thread caps occupancy the same way.
A cache-resident benchmark can report effective bandwidth above the HBM datasheet without a timer bug. For an HBM experiment, use a working set larger than L2 or an explicit eviction protocol, and confirm DRAM traffic with counters. For production behavior, reproduce the actual cache state and predecessor kernels. Report which question the benchmark answers. The Nsight Compute Profiling Guide, checked October 17, 2026, describes how replay and cache control change the experiment.
What interviewers listen for
They listen for capacities with units and for the word reuse. "Shared memory is fast" is a slogan; "228 KB per SM, so a 128 by 128 BF16 tile is 32 KiB and I can double-buffer K and V" is an engineer. Strong candidates know the model never fits in L2, can say why decode at batch 1 is bound by HBM regardless of kernel quality, and catch the L2-resident benchmark.
Say it in 30 seconds
An H100 SXM has about 33 MiB of registers, up to 29 MiB of shared memory, a 50 MB L2 and 80 GB of HBM at 3.35 TB/s. A single Llama 8B layer is 436 MB, so weights always stream from HBM and decode at batch 1 is floored at 4.8 ms per token. Kernels win by loading a byte once and reusing it from shared memory and registers, and benchmarks lie when the test tensor fits in L2.