TL;DR: A batch-1 decode step does 2 FLOPs per weight it reads, an intensity of 1 FLOP/byte in BF16 against an H100 ridge of about 295, so the step is priced in bytes. Each of two tensor-parallel H100s streams 70.6 GB of weights per token, which at 3.35 TB/s is a 21.1 ms floor, about 47 tokens per second, before KV reads, communication and launch gaps are added.
How to approach it
Start by stating the work per step, not the hardware: one new token's activation vector is multiplied by every weight matrix, so the step is a chain of matrix-vector products. Then ask the questions that move the answer: what precision are the weights in, what is the tensor-parallel degree, how long are the contexts, and is the question about per-sequence latency or aggregate throughput. Only then reach for the datasheet.
A strong answer
A matrix-vector product reads each weight once and does one multiply and one add with it. With one sequence in the batch, the step's arithmetic intensity is FLOPs over bytes moved:
where is the parameter count, the batch and bytes per weight. In BF16 that is exactly FLOP/byte, so 1 at batch 1. The H100 SXM's dense BF16 peak over its HBM bandwidth puts the ridge, where a kernel stops waiting on memory, two orders of magnitude higher:
ridge = 989 TFLOP/s / 3.35 TB/s = 295 FLOP/byte
batch-1 decode intensity = 1 FLOP/byte -> 295x below the ridge: bandwidth-bound
So the floor is the time to stream the weights. Llama 3.3 70B has 70.55B parameters from its config (80 layers, hidden 8192, untied embeddings). Tensor parallelism splits every matrix across the two GPUs, and both read their halves at the same time:
weights = 70.55e9 params x 2 B = 141.1 GB
per GPU (TP=2) = 70.55 GB
floor = 70.55e9 B / 3.35e12 B/s = 21.06 ms per token -> 47.5 tok/s
compute time = 2 x 70.55e9 / 2 GPUs / 989e12 = 0.071 ms (300x smaller)
sanity = TP=4 halves the bytes per GPU: 10.5 ms. FP8 weights do the same.
| Configuration | Bytes per GPU per step | Floor |
|---|---|---|
| 2 x H100, BF16 | 70.6 GB | 21.1 ms |
| 2 x H100, FP8 weights | 35.3 GB | 10.5 ms |
| 4 x H100, BF16 | 35.3 GB | 10.5 ms |
| 2 x H200, BF16 (4.8 TB/s) | 70.6 GB | 14.7 ms |
| 1 x B200, FP8 (8 TB/s) | 70.6 GB | 8.8 ms |
Three things keep a real server above the floor, and naming them with magnitudes is what separates the answer:
- KV reads. Attention also streams the sequence's cache. At 8,192 tokens, Llama 70B's cache is 2.5 GiB (320 KiB per token), 1.34 GB per GPU at TP=2, which adds about 0.4 ms. Negligible at batch 1, dominant at large batch and long context. Decode attention never gets cheaper with batching, because every sequence brings its own cache: its intensity is query heads over KV heads, FLOP/byte, whatever the batch.
- Communication. Megatron-style TP does two all-reduces of the hidden state per layer, 160 per token. The payload is tiny (16 KB in BF16), so each one costs latency, not bandwidth. If each takes 10 microseconds, that is 1.6 ms, already 8% of the floor, and the count does not shrink when you add GPUs.
- Launch and scheduling overhead. A decode step is on the order of a thousand small kernels. Launched one by one from Python, the CPU becomes the bottleneck long before HBM does, which is why every serious engine captures decode into CUDA graphs.
The measured number to report is model bandwidth utilization: bytes the step had to read divided by measured step time, over peak bandwidth. A 70B TP=2 step measured at 26 ms has an MBU of . That one ratio tells you whether to look for a kernel problem or accept the hardware's verdict.
The reversal condition: the floor stops being the right model when the batch grows. Decode at batch reads the same 70.6 GB but does times the arithmetic, so per-step time stays near 21 ms while aggregate throughput rises almost linearly, until weight intensity reaches the ridge near , or, more often in practice, until KV memory runs out first.
What interviewers probe next
"Now serve 64 sequences at 4K context on the same two GPUs." It does not fit: 64 x 4,096 tokens x 320 KiB is 85.9 GB of KV, more than the FP8 weights. At 32 sequences with FP8 weights each GPU reads 35.3 GB of weights plus 21.5 GB of KV per step, about 16.9 ms, roughly 1,900 tokens per second in aggregate.
"Why does TP=8 not give four times lower latency than TP=2?" The weight bytes per GPU fall 4x, but the 160 all-reduces per token do not, and their latency is fixed per call. The floor drops from 21.1 ms to 5.3 ms while the communication share grows from a rounding error to a large fraction of the step.
"Does INT4 weight-only quantization halve it again?" In bytes, yes: 17.6 GB per GPU at TP=2, a 5.3 ms floor. It only lands if the dequantizing GEMV kernel still runs near peak bandwidth, which is what Marlin-class kernels exist to do.
"What would you put on the dashboard?" Per-step time split into weight reads, KV reads, communication and idle, plus MBU. Tokens per second alone cannot tell a slow kernel from a saturated GPU.
How it's scored
Weak: divides FLOPs by peak TFLOP/s and quotes a 0.07 ms step, or uses the 1,979 TFLOP/s sparse figure, or forgets that TP splits the weights.
Hire: derives intensity of about 1 FLOP/byte against a ridge near 295, produces the 21 ms floor from bytes over bandwidth, and explains why batching raises throughput without raising step time.
Strong Hire: adds the KV term and shows that decode attention intensity is fixed at query-to-KV head ratio, quantifies communication and launch overhead, and defines MBU as the metric that separates a kernel problem from physics.
Common mistakes
- Mixing GB and GiB: the 141 GB of weights is 131.4 GiB, and a 7% error compounds through every later step.
- Quoting tokens per second without saying whether it is per sequence or aggregate. At batch 32 those differ by 32x.
- Assuming more GPUs always lower latency. Past the point where per-GPU weight reads are a few milliseconds, all-reduce latency and launch gaps set the floor.
- Treating 3.35 TB/s as achievable. It is the datasheet peak; the gap between it and measured bandwidth is the whole point of reporting MBU.