Skip to content
← Inference System Design / 14
expertNewOpenAIAnthropicMeta

Plan capacity for a 70B fleet that mixes H100 and B200 nodes under one latency SLO. How do you count capacity and route across them?

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackDesign batch inference over 1 billion documents: embed each one and run a 70B model over each. How many GPUs, how long, what cost?Next in this trackCut inference cost per million output tokens by 70% while preserving quality and latency. Can you defend the target?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.