Skip to content
← Inference Physics & Roofline / 16
hardNewNVIDIATogether AIFireworks AI

Plot tokens per second per GPU against tokens per second per user for a 70B. Where does that frontier come from?

Every serving benchmark now plots one curve: GPU efficiency against user speed. The answer derives that curve from bytes and overheads, shows why tensor parallelism is nearly free in the roofline and expensive in practice, and explains why a single benchmark point cannot rank two GPUs.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Every serving benchmark now plots one curve: GPU efficiency against user speed. The answer derives that curve from bytes and overheads, shows why tensor parallelism is nearly free in the roofline and expensive in practice, and explains why a single benchmark point cannot rank two GPUs.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackWe moved prefill GEMMs from FP8 to NVFP4 on a B200, and 32K-token TTFT improved 1.4x, not 2x. Explain the gap.Next in this trackServe a 70B on TPU7x Ironwood: give the roofline, explain ragged paged attention, and justify vLLM's small token budget.Next in this trackA distillation scorer runs out of memory on known student continuations. Do you need another decode engine?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.