The Roofline Model for LLM Inference

TL;DR: A kernel can run no faster than the smaller of peak FLOP/s and bandwidth times its FLOPs per byte. An H100 SXM has a dense BF16 ridge of 989 TFLOP/s / 3.35 TB/s = 295 FLOP/byte. Batch-1 linear layers sit near 1 FLOP/byte, but a complete decode step also pays for attention's KV reads. Bound each dependent phase, add the times, and check that the batch fits in memory before quoting throughput.

The bound

The roofline model, from Williams, Waterman and Patterson (2009), asks one question of a kernel: how many FLOPs does it do per byte it moves from main memory? Call that the arithmetic intensity $I$. The attainable throughput is then capped by two lines:

Left of the ridge, the slanted line binds: the tensor cores sit idle waiting on HBM, and the kernel's time is bytes over bandwidth. Right of it, the flat line binds: memory keeps up and time is FLOPs over peak. The ridge itself is $P_{\text{peak}} / BW$, a property of the chip and the precision, not of the model.

Where the inference kernels land

A linear layer multiplies a $[T, d_{in}]$ activation by a $[d_{in}, d_{out}]$ weight. It does $2 T d_{in} d_{out}$ FLOPs and, when $T$ is small, its bytes are dominated by the weight, $d_{in} d_{out} b$. So intensity is about $2T/b$: in BF16, one FLOP per byte per token in the step. During decode, $T$ is the batch. During prefill, $T$ is the batch times the prompt length.

For dense grouped-query attention, each sequence reads its own keys and values. With BF16 KV and ideal reuse across query heads, the dominant attention matmuls have intensity equal to the query-to-KV head ratio, independent of batch. This approximation excludes softmax, activation traffic and cache effects.

Worked example: Llama 3.1 8B on one H100

Use the Llama 3.1 8B configuration: 32 layers, width 4,096, intermediate width 14,336, 32 query heads, 8 KV heads, head dimension 128 and vocabulary 128,256. All sizes below use decimal GB. Hardware peaks are from NVIDIA's H100 specifications, with the sparsity multiplier removed.

matrix params/layer = 2 x 4096² + 2 x 4096 x 8 x 128 + 3 x 4096 x 14336
                    = 218,103,808
P_linear            = 32 x 218,103,808 + 128256 x 4096 = 7,504,658,432
BF16 matrix bytes   = 2 x P_linear = 15.0093 GB
KV bytes/token      = 2 (K,V) x 32 layers x 8 heads x 128 x 2 bytes = 131,072

batch B = 8, cached context S = 8,192 tokens per sequence, TP = 1
linear time         = max(15.0093e9 / 3.35e12, 2 x P_linear x 8 / 989e12)
                    = max(4.4804, 0.1214) ms = 4.4804 ms
attention bytes     = 8 x 8192 x 131072 = 8.5899 GB
attention FLOPs     = 4 x 32 x 8 x 8192 x 32 x 128 = 34.3597 GFLOPs
attention time      = max(8.5899e9 / 3.35e12, 34.3597e9 / 989e12) = 2.5642 ms
step floor          = 4.4804 + 2.5642 = 7.0446 ms
aggregate ceiling   = 8 / 0.0070446 = 1,136 tokens/s
per-sequence ceiling = 1 / 0.0070446 = 142 tokens/s

The input embedding is a lookup, so it is excluded from the full matrix scan; the output vocabulary projection is included. Both consume storage. Adding input embeddings and normalization parameters gives 8,030,261,248 stored parameters, or 16.0605 GB in BF16.

At the same context, batch 32 raises the KV term to 10.2566 ms and the total to 14.7370 ms: 2,171 tokens/s aggregate, but only 68 per sequence. At batch 64 the memory check fails under a chosen 90% H100 budget with 4 GB reserved for runtime: 16.0605 GB weights + 64 × 8,208 × 131,072 bytes paged KV + 4 GB = 88.91 GB, above 72 GB. The 8,208-token allocation rounds the context plus one output slot to 16-token blocks. A hypothetical time for that batch is not an attainable throughput.

Change the batch and context in the inference lab. The model sums dependent linear and attention phase bounds; taking one global maximum can hide a slow phase behind another phase's unused capacity. It excludes launch gaps, communication, activation traffic and KV writes. The inference scaling chapter develops the same phase-by-phase reasoning.

Precision moves both axes

Halving the bytes per weight doubles intensity, and on Hopper the FP8 tensor-core peak is also double the BF16 one, so the ridge doubles to 591. A W8A8 decode step keeps the same crossover batch of about 295 while every step reads half the bytes. Weight-only FP8 with BF16 math is different: intensity doubles but the ridge stays at 295, so the crossover falls to a batch of about 148. On an H200 the ridge is lower (206 in BF16) because bandwidth grew to 4.8 TB/s with the same compute, so decode reaches compute-bound earlier there. Sparse datasheet peaks double the ridge on paper; dense LLM weights never see them.

Where it breaks

The roofline is a bound, and real kernels miss it for reasons it does not model. Small GEMMs (a few dozen rows) underfill the tensor cores, so measured compute peaks well below 989. Launch gaps between hundreds of short decode kernels show up as time with no bytes moving at all; that is latency-bound, a third regime the two lines cannot express. Tensor-parallel all-reduces add fixed per-layer latency. Caches break the byte count: a model whose weights fit in L2 is not reading HBM. And intensity assumes each byte moves once; a kernel that re-reads tiles from HBM because its tiling is poor has a higher effective byte count than the arithmetic says.

What interviewers listen for

They want the ridge computed from the dense peak, with the part and precision named. Quoting 1,979 TFLOP/s for BF16 is the sparse figure and doubles the ridge. They listen for intensity derived from shapes ($2T/b$) rather than asserted. Strong candidates separate the weight term, which batching amortizes, from the attention term, which it never does, and name the third regime when the step is too short for either line to bind.

Say it in 30 seconds

"Compute a roofline for each phase. H100 SXM's dense BF16 ridge is about 295 FLOPs per byte; small decode linear layers sit below it. Batching amortizes the weight read, but each sequence adds KV reads. Add the dependent phase times and check memory capacity. Higher aggregate throughput can coexist with worse latency for every sequence."