Skip to content
← Quantization & Numerics / 12
mediumNewNVIDIAMetaBaseten

Fit a 70B model on one 80 GB H100 with room left for KV cache. Choose the weight and KV formats and show the arithmetic.

FP8 halves the weights and still leaves almost no memory for the cache. The answer does the budget line by line, picks the format pair that leaves room for real concurrency, and says which hardware change makes the opposite choice correct.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

FP8 halves the weights and still leaves almost no memory for the cache. The answer does the budget line by line, picks the format pair that leaves room for real concurrency, and says which hardware change makes the opposite choice correct.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackYour serving team does not own pretraining. When is quantization-aware training worth it over post-training quantization?Next in this trackA quantized MoE holds its benchmark scores but loses accuracy on rare domains, traced to rarely routed experts. Explain why and fix it.Next in this trackDesign the numerics policy for a fleet that spans Hopper and Blackwell: formats per generation, fallbacks, and how you validate them.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.