hardNewOpenAIAnthropicNVIDIA

An 8B model at batch 1 on an H100 should take 4.8 ms per step but takes 12 ms. Name the three likeliest causes, with numbers.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackRank the decode-time arithmetic intensity of the QKV projection, attention, the MLP and the LM head. Which ones can fusion help?Next in this trackDesign a roofline-based performance model that predicts TTFT and TPOT for any model, GPU and batch. Where will it be wrong?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.