Skip to content
← Inference Physics & Roofline / 17
hardNewNVIDIAOpenAITogether AI

We moved prefill GEMMs from FP8 to NVFP4 on a B200, and 32K-token TTFT improved 1.4x, not 2x. Explain the gap.

Doubling tensor-core throughput halves only the part of prefill that runs on the GEMMs. The answer splits a long prompt into its linear and attention phases, finds the third roof that sets the attention floor on Blackwell, and says when FP4 is worth the quality risk.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Doubling tensor-core throughput halves only the part of prefill that runs on the GEMMs. The answer splits a long prompt into its linear and attention phases, finds the third roof that sets the attention floor on Blackwell, and says when FP4 is worth the quality risk.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackServe a 70B on TPU7x Ironwood: give the roofline, explain ragged paged attention, and justify vLLM's small token budget.Next in this trackA distillation scorer runs out of memory on known student continuations. Do you need another decode engine?Next in this trackA 0.6B policy's rollout step takes 22 s for 256 answers on a 273 GB/s device. What bounds it, and what shortens it?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.