GPU performance chains: bytes in flight, occupancy, tiles, waves and gaps

These chains use the H100 SXM throughout: 132 SMs, 3.35 TB/s of HBM, 989 TFLOP/s dense BF16, 65,536 registers and 228 KB of shared memory per SM. A kernel question usually turns on one of them.

Bytes in flight to reach bandwidth

inputs   3.35 TB/s of HBM bandwidth, an assumed 600 ns of memory latency
formula  bytes in flight = bandwidth x latency (Little's law for memory)
GPU      3.35e12 B/s x 600e-9 s = 2.0 MB outstanding at all times
per SM   2.0 MB / 132 = 15 KB per SM
sanity   a kernel with fewer loads in flight than that cannot reach peak,
         however simple it is

Decode GEMVs fall short in three ways: too few thread blocks for 132 SMs, loads too narrow per thread, and matrices so small that launch gaps rival the kernel.

Drilled in: Decode GEMV at half of HBM bandwidth · Bytes in flight and Little's law

A coalesced warp load

inputs   row-major [N, K] BF16 weight, K = 4,096, so a row is 8,192 B
coalesced  32 lanes walk along one row, 16 B each: one instruction reads
         512 contiguous bytes, 16 full 32-byte sectors
row per lane  lanes sit 8,192 B apart: 32 separate sectors per instruction,
         each only half used
sanity   same bytes requested, twice the sectors fetched, and far fewer
         bytes per request in flight

Consecutive lanes must read consecutive addresses. For nn.Linear's layout that means lanes along K and a warp reduction at the end.

Drilled in: GEMV coalescing, row-major against transposed

Occupancy from three limits

inputs   256-thread blocks, 128 registers per thread, 48 KB of shared memory
         per block; H100 (compute capability 9.0)
registers  65,536 / (256 x 128) = 2 blocks per SM
shared   228 KB / (48 + 1) KB = 4 blocks (CUDA reserves 1 KB per block)
warps    64 warps per SM / 8 per block = 8 blocks
result   registers bind: 2 blocks, 16 of 64 warps, 25% occupancy
sanity   64 registers per thread doubles it to 50%, where shared memory binds

Higher occupancy helps a latency-bound kernel hide memory latency. A kernel that already keeps enough bytes in flight gains nothing from it.

Drilled in: Occupancy at 128 registers and 48 KB

Shared-memory bank conflicts

inputs   32 banks, 4 bytes wide; float tile[32][32]
rule     word w lives in bank w mod 32
column   lane r reads tile[r][c], word 32r + c: every lane hits bank c,
         a 32-way conflict, 32 wavefronts
padded   rows of 33 words: word 33r + c sits in bank (r + c) mod 32,
         distinct for every lane, one wavefront
sanity   one extra word per row removes a 32x slowdown on column reads

Padding stops working when rows must stay 16-byte aligned for vector or TMA stores. A swizzled layout is the fix then.

Drilled in: Bank conflicts and transpose padding

GEMM tile intensity

inputs   output tile BM x BN, K-step BK, BF16 operands
per step loads (BM + BN) x BK x 2 B, computes 2 x BM x BN x BK FLOPs
intensity  BM x BN / (BM + BN) FLOP per byte
128 x 128  64;  128 x 256  85
limits   a 128 x 256 FP32 accumulator is 32,768 registers, half the SM's
sanity   bigger tiles cut operand traffic until registers or the 227 KB
         shared-memory limit per block stops them

Tile shape sets how often operands are re-read from L2 and HBM. It is the first knob in a GEMM and the one the hardware bounds hardest.

Drilled in: Tensor-core GEMM tile sizes on Hopper

Wave quantization

inputs   140 output tiles, one resident per SM, 132 SMs
waves    ceil(140 / 132) = 2; the second runs 8 tiles while 124 SMs idle
efficiency  140 / 264 = 53%
time     about 143 us against the 76 us its FLOPs need at 989 TFLOP/s
sanity   a 512-token prefill chunk makes 112 tiles and fits in one wave

Stream-K splits the K loop across SMs to fill the second wave. The scheduler can often avoid it more cheaply by choosing chunk sizes that land on whole waves.

Drilled in: Wave quantization, 140 tiles on 132 SMs

Fused RMSNorm traffic

inputs   an 8,192-token prefill chunk, hidden 8,192, BF16
fused    one read and one write per element, 4 B: 268 MB, 80 us at 3.35 TB/s
eager    eight kernels with FP32 intermediates, about 36 B per element:
         2.42 GB, 0.72 ms
sanity   9x the bytes for the same arithmetic; at decode batch 1 the bytes
         are trivial and the eight launches are the cost

Fusion matters for memory-bound elementwise work in prefill and for launch count in decode, for different reasons.

Drilled in: Fused RMSNorm in Triton

Launch gaps in a decode step

inputs   an illustrative Llama 3.1 8B decode step on H100 SXM:
         325 kernels running for 5.2 ms in total, step time 7.8 ms
gaps     1.8 ms of small uniform gaps between kernels (launch-bound)
stall    0.8 ms in one gap at the step boundary (a host synchronization)
MBU      16.06 GB / 7.8 ms / 3.35 TB/s = 62%
sanity   the kernels ran at 92% of bandwidth; the step lost a third of its
         time between them

Many small gaps call for CUDA graphs. One large gap calls for finding the synchronization and the CPU work behind it.

Drilled in: Nsight Systems: launch gaps in a decode step · CUDA graphs for decode