The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.
an account raises the free count per topic
← Inference Physics & Roofline / 06
mediumNewNVIDIAOpenAITogether AI
We are moving a 70B model from H100s to B200s. How do decode latency and throughput change, regime by regime?
The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackDeepSeek V3 has 671B parameters but only 37B active. Why does a decode step at batch 64 read far more than 37B?Next in this trackHow would you measure MBU on a running server? What do you time, what do you count, and which numbers make the result wrong?Next in this trackYour decode step runs at 60% of the bandwidth floor. Where does the rest of the time go? Build me the budget.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.