hardNewNVIDIAFireworks AITogether AI

In prefill-decode disaggregation, how does KV transfer size scale, and what link bandwidth keeps the transfer off the critical path?

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackA model's config says max context 1M. Size the KV cache for a single 1M-token request on a GQA model and on an MLA model.Next in this trackWhen you quantize the KV cache, why are keys more sensitive than values, and what per-channel or per-token scheme follows from that?Next in this trackDesign a multi-tier KV cache across HBM, host DRAM and SSD for a fleet with heavy prefix reuse, and show the hit-rate math.

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.