Skip to content
← Distributed & Disaggregated Inference / 18
hardNewNVIDIATogether AIFireworks AI

In prefill/decode disaggregation, should decode pull the KV or prefill push it? What does each choice cost?

Most candidates argue bandwidth, and bandwidth barely enters into it: the transfer is about 2% of prefill time for a 70B model. The answer finds what push and pull really trade, which side holds KV memory while the other side queues, and when that flips the choice.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Most candidates argue bandwidth, and bandwidth barely enters into it: the transfer is about 2% of prefill time for a 70B model. The answer finds what push and pull really trade, which side holds KV memory while the other side queues, and when that flips the choice.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackShould a vision-language model's image encoder run on its own instances? Size the encoder work and its cache, then decide.Next in this trackAttention-FFN disaggregation for MoE decode: why ping-pong micro-batches, how many, and when does it beat wide EP?Next in this trackGrow and shrink a running expert-parallel MoE deployment without a restart: what moves, what pauses, what does it cost?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.