Distributed inference chains: all-reduces, all-to-alls, transfers and bubbles

Every parallelism question comes down to which bytes cross which link, how often, and whether that traffic sits on the critical path of a step. The chains below count the bytes; the linked questions decide what to do about them.

Two all-reduces per layer

inputs   Llama 3.3 70B: 80 layers, hidden 8,192, BF16 activations; TP=8
         Megatron split: column-parallel first matrix, row-parallel second,
         one all-reduce after attention and one after the MLP
calls    2 x 80 = 160 all-reduces per step
message  8,192 x 2 B = 16,384 B per token per all-reduce
ring     each GPU sends 2 x (8 - 1) / 8 x 16,384 = 28,672 B per call
total    160 x 28,672 B = 4.59 MB sent per token per GPU
sanity   the bytes are small; at decode the 160 calls cost more in latency
         than in bandwidth

The call count is set by the layer count, not the TP degree. Going from TP=4 to TP=8 halves each GPU's weights and keeps all 160 calls.

Drilled in: Megatron column and row splits

What TP=8 does to a 70B step

inputs   Llama 3.3 70B BF16, 141.1 GB of weights, H100 SXM 3.35 TB/s, batch 1
         an assumed 10 us per all-reduce, 160 per step
TP=4     141.1 / 4 / 3.35 = 10.5 ms of reads + 1.6 ms of all-reduce = 12.1 ms
TP=8     141.1 / 8 / 3.35 = 5.3 ms of reads + 1.6 ms of all-reduce = 6.9 ms
gain     12.1 / 6.9 = 1.77x from twice the GPUs
sanity   the fixed 1.6 ms grows from 13% to 23% of the step, and to 38%
         at TP=16

The 10 µs is an assumption, and the conclusion is sensitive to it. Measure the per-call latency on the node before choosing between one TP=8 replica and two TP=4 replicas.

Drilled in: TP=8 or two TP=4 replicas

MoE all-to-all bytes

inputs   DeepSeek V3: 58 MoE layers, hidden 7,168, top-8 routing
         dispatch in FP8 (1 B), combine in BF16 (2 B)
dispatch 8 x 7,168 x 1 B = 57,344 B per token per layer
combine  8 x 7,168 x 2 B = 114,688 B per token per layer
per token (57,344 + 114,688) x 58 = 9.98 MB
batch 256  256 x 9.98 MB = 2.55 GB per step
sanity   at EP=8 inside one node, 7 of every 8 expert copies leave the GPU

Placement changes the share that crosses the slowest link, not the total. Across nodes, sending each token once per destination node instead of once per expert is what cuts the network share.

Drilled in: Expert-parallel all-to-all bytes · Expert parallelism and all-to-all

Does 405B fit on one node

inputs   Llama 3.1 405B; one node of 8 x H100 SXM 80GB
BF16     811.7 GB of weights / 8 = 101 GB per GPU: does not fit in 80 GB
FP8      405.9 GB / 8 = 50.7 GB per GPU: fits, and 78.22 - 50.7 = 27.5 GB
         per GPU is left for KV and activations at vLLM's default budget
sanity   BF16 needs a second node, and with it pipeline parallelism or
         TP=16 across the slower inter-node link

The fit check comes before any latency comparison. Once both layouts fit, the comparison is about the per-token path, and pipeline stages add their times one after another.

Drilled in: 405B on one node or two

KV transfer for disaggregation

inputs   Llama 3.3 70B BF16, 320 KiB of KV per token, an 8,192-token prompt
         prefill at TP=4 on H100 SXM; one 400 Gb/s (50 GB/s) NIC per GPU
KV       8,192 x 327,680 B = 2.68 GB (2.5 GiB)
transfer 2.68 GB / (4 x 50 GB/s) = 13.4 ms
prefill  2 x 70.55e9 x 8,192 / (4 x 989e12) = 292 ms at peak
sanity   4.6% of the prefill that produced it; over one NIC it is 54 ms

Each TP rank holds its own share of the KV heads, so the four NICs carry the transfer in parallel only if the connector sends per rank. A decode pool at a different TP degree has to reshard heads on arrival.

Drilled in: Disaggregation KV transfer bandwidth · Prefill-decode disaggregation

DP attention capacity for MLA

inputs   DeepSeek V3 on 8 x H200 SXM: FP8 weights, BF16 MLA KV 70,272 B per token
         vLLM's 0.92 of 139.80 GiB = 138.1 GB usable per GPU
TP=8     weights 671.0 / 8 = 83.9 GB, KV 54.2 GB per GPU, replicated:
         54.2 GB / 70,272 B = 772k tokens for the node
DP attn  weights (671.0 - 11.4) / 8 + 11.4 = 93.9 GB, KV 44.2 GB per GPU, each
         rank its own: 8 x 44.2 GB / 70,272 B = 5.04M tokens
sanity   6.5x the capacity for 10 GB more weights per GPU

MLA's latent is shared by every head, so splitting heads across ranks splits no KV. A GQA model with enough KV heads gets its cache divided by TP and has much less reason to switch.

Drilled in: DP attention plus EP for MLA

The pipeline bubble

inputs   p pipeline stages, m micro-batches per fill and drain
formula  idle fraction = (p - 1) / (m + p - 1)
p 4, m 4   3 / 7 = 43%
p 4, m 16  3 / 19 = 16%
latency  a token still visits all p stages in turn: per-token time is the sum
         of the stage times
sanity   pipeline stages buy capacity and throughput, never lower latency

At decode, keeping m at least p means keeping that many independent batches in flight, which needs the KV room for all of them.

Drilled in: Pipeline bubble and latency