mediumNewGoogle DeepMindAnthropicMeta

A model's config says max context 1M. Size the KV cache for a single 1M-token request on a GQA model and on an MLA model.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackWhen you quantize the KV cache, why are keys more sensitive than values, and what per-channel or per-token scheme follows from that?Next in this trackDesign a multi-tier KV cache across HBM, host DRAM and SSD for a fleet with heavy prefix reuse, and show the hit-rate math.Next in this trackA hybrid model mixes attention and recurrent layers. Budget its state and design safe prefix reuse and rollback.

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.