Skip to content
Practice tests · 30 items

Inference Physics & Roofline practice test

Bytes and FLOPs per token, why batch-1 decode reads every weight once per step, the ridge point and the batch size that crosses it, MBU versus MFU, and the millisecond latency floors you must produce before anyone draws a box. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test
Topic
Difficulty
Questions
10 in this pool · about 7 min
Reveal answers
Sign in to startNeeds a free account · retakes deal unseen items first

Sample items with answers

easy · sample
Llama 3.3 70B has 70.554 billion parameters. How much memory do its weights take in BF16, and what does that say about a single H100 SXM 80 GB?
About 70.6 GB, so it fits on one H100 with roughly 9 GB left for KV cache
✓About 141 GB, so the weights alone need at least two 80 GB GPUs
About 131 GB, because the 141 figure double-counts the embedding table
About 282 GB, because BF16 checkpoints store a master copy beside each value

BF16 is 2 bytes per parameter: 70.554e9 x 2 B = 141.1 GB, which is 131.4 GiB. The 131 option is the same bytes quoted in GiB with the unit swapped, not a different count, and the embedding table is part of the 70.554B. The 70.6 GB figure is the FP8 size, the usual slip of counting one byte per parameter. Master copies belong to training. So BF16 70B needs at least two 80 GB parts before a single token of KV cache, which is why TP=2 is the smallest sensible BF16 deployment.

easy · sample
Why is batch-1 decode on a dense transformer bandwidth-bound on an H100 SXM rather than compute-bound?
Attention over the KV cache needs far more FLOPs than the weight matrix multiplies do
The GPU runs decode on CUDA cores, because tensor cores need a batch of at least 64 rows
✓Each step reads every weight once to do about 2 FLOPs per 2-byte weight, near 1 FLOP/byte
Sampling the next token synchronizes with the host every step, which caps how fast the GPU runs

At batch 1 each BF16 weight (2 bytes) feeds one multiply-add (2 FLOPs) per step, so arithmetic intensity is about 1 FLOP per byte. The H100 SXM ridge is 989 TFLOP/s dense BF16 over 3.35 TB/s, about 295 FLOP/byte, so the tensor cores sit idle waiting on HBM. Tensor cores do run at batch 1 through padded tiles, they just have nothing to do. Attention FLOPs at batch 1 are small next to the weight GEMVs, and host sync is a latency overhead, not the reason the step is bound by bytes.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: