Skip to content
← Distributed & Disaggregated Inference / 11
hardNewGoogle DeepMindMetaNVIDIA

A 128k-token prefill takes 10 seconds on one node. How does context parallelism split it across nodes, and what does it send?

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackA multi-node MoE deployment's TPOT doubled overnight, and one network link is degraded. How do you find it, and how do you design around it?Next in this trackEach decode layer needs the previous all-to-all to finish. How do you overlap communication with compute anyway, and at what cost?Next in this trackYou need to get 500 GB of weights onto 1,000 GPUs as fast as possible. What is the lower bound, and what design gets close to it?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.