hardNewOpenAIAnthropicFireworks AI

The KV pool is full and requests keep arriving. Compare admission control, preemption and spilling, and what each does to p99.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackIn prefill-decode disaggregation, how does KV transfer size scale, and what link bandwidth keeps the transfer off the critical path?Next in this trackA model's config says max context 1M. Size the KV cache for a single 1M-token request on a GQA model and on an MLA model.Next in this trackWhen you quantize the KV cache, why are keys more sensitive than values, and what per-channel or per-token scheme follows from that?

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.