Skip to content
← Distributed & Disaggregated Inference / 06
hardNewNVIDIAPerplexityFireworks AI

You split prefill and decode onto separate nodes. Compute the KV transfer per request and the network bandwidth it needs.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackWhy does a one-shot custom all-reduce beat NCCL for small decode messages on an NVLink node, and when does it lose?Next in this trackServe Llama 3.1 405B: TP=8 on one H100 node, or TP=4 times PP=2 across two nodes? Work the latency and the memory.Next in this trackSome experts get far more tokens than others. What do hot experts do to step time, and how do redundant experts and rebalancing fix it?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.