Skip to content
← Distributed & Disaggregated Inference / 08
hardNewMetaNVIDIATogether AI

Serve Llama 3.1 405B: TP=8 on one H100 node, or TP=4 times PP=2 across two nodes? Work the latency and the memory.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackSome experts get far more tokens than others. What do hot experts do to step time, and how do redundant experts and rebalancing fix it?Next in this trackA GB200 NVL72 puts 72 GPUs in one NVLink domain. What does that change for MoE serving and for prefill-decode disaggregation?Next in this trackA 128k-token prefill takes 10 seconds on one node. How does context parallelism split it across nodes, and what does it send?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.