Pipeline parallelism for inference: why does it raise throughput but never lower per-token latency? Work the bubble arithmetic.

TL;DR: A token traverses the stages one after another, so pipeline parallelism leaves per-token latency at the sum of the stage times: 30.3 ms for Llama 3.1 405B in BF16 split over two 8-GPU H100 nodes. What it buys is capacity and throughput, provided at least $p$ micro-batches are in flight; a batch that fills and drains wastes $(p-1)/(m+p-1)$ of the time, 50% for two stages and one micro-batch.

How to approach it

Separate the two questions first: how long one sequence waits per token (latency), and how many tokens the GPUs produce per second (throughput). Pipelining only touches the second. Ask whether the model fits in one NVLink domain, because that is usually why PP is on the table at all, and ask the inter-node all-reduce latency, because the real alternative is tensor parallelism stretched across nodes.

A strong answer

Latency. Llama 3.1 405B is 811.7 GB of BF16 weights, more than an 8-GPU H100 node holds. Split it as TP=8 within each node and PP=2 across two nodes: stage 1 runs layers 1 to 63, stage 2 runs 64 to 126, and each GPU holds one sixteenth of the weights. In the bandwidth-bound decode regime a stage costs its weight read:

weights per GPU  = 811.7 GB / 16 = 50.7 GB
one stage        = 50.7 GB / 3.35 TB/s = 15.14 ms   (weights only, KV excluded)
one token        = stage 1 + hop + stage 2 = 15.14 + 0.04 + 15.14 = 30.3 ms
hop              = 64 seqs x 16,384 x 2 B = 2.1 MB over a 50 GB/s NIC = 42 us

The pipeline added one cheap hop and nothing else. It also removed nothing: the same 811.7 GB are read per token, just by two groups in turn. That is why PP never lowers latency below what one group holding every layer would achieve.

Throughput and the bubble. While stage 2 works on micro-batch A, stage 1 is idle unless micro-batch B exists. If a fixed batch is cut into $m$ micro-batches that enter and drain (one big prefill, or a synchronous step), the idle fraction is:

$$\text{bubble} = \frac{p-1}{m+p-1}$$

p = 2:  m = 1 -> 1/2 = 50.0%   m = 2 -> 1/3 = 33.3%   m = 4 -> 20.0%   m = 8 -> 11.1%
p = 4:  m = 1 -> 3/4 = 75.0%   m = 2 -> 60.0%         m = 4 -> 42.9%   m = 8 -> 27.3%

In continuous decoding the pipeline need not drain. Keep $p$ independent micro-batches cycling and each stage is always busy:

time      0-15 ms     15-30 ms    30-45 ms    45-60 ms
stage 1   A tok t     B tok t     A tok t+1   B tok t+1
stage 2   idle        A tok t     B tok t     A tok t+1

The catch is that in the bandwidth-bound regime each micro-batch pays for its own weight read. Two micro-batches of 64 produce 128 tokens every 30.3 ms, 4,226 tok/s. To match a single step of batch 128 you must hold 256 sequences in flight (two micro-batches of 128), which gives 8,452 tok/s at the same 30.3 ms per token. Little's law: PP multiplies the concurrency, and the KV memory, needed for a given throughput by roughly $p$.

Against TP=16 across the two nodes. Cross-node TP divides the weight read by 16 but sends 252 all-reduces per token over the network. With an assumed fixed cost $\alpha$ per inter-node all-reduce (a measured quantity, not a datasheet one):

TP=16, batch 1: 15.14 ms + 252 x alpha
  alpha = 30 us -> 23.0 ms   (beats PP's 30.3 ms)
  alpha = 60 us -> 30.6 ms   (break-even alpha = 15.14 ms / 252 = 60 us)
TP=16, batch 128: each all-reduce moves 2 x 15/16 x 4.19 MB = 7.9 MB at 50 GB/s
  = 157 us of bandwidth + 30 us -> 252 x 187 us + 15.14 ms = 62.3 ms per step
  = 2,053 tok/s, against PP's 4,226 at the same 128 sequences

So the decision: PP across nodes for throughput-priced traffic and whenever the inter-node all-reduce costs more than about 60 µs; cross-node TP only for a batch-1 latency target on a fabric fast enough to beat that threshold. And if the model fits in one node (FP8 weights, or H200s with 141 GB each), neither is needed and a single TP=8 group wins on both counts.

What interviewers probe next

"Prefill a 32k-token prompt with PP=4." Chunk it into $m$ pieces so the stages overlap. With 8 chunks the bubble is 27.3%, and the time to first token is still a full traversal of the last chunk on top of the pipelined rest.

"Does PP help the KV cache?" Each stage stores KV only for its own layers, so per-GPU KV per token halves at PP=2, but you need $p$ times the sequences in flight to keep the stages busy, which spends that saving.

"Why does PP pair well with Ethernet-class interconnect?" It sends one activation per token per stage boundary, 32 KB here, instead of 252 all-reduces. It is the parallelism that tolerates a slow link.

How it's scored

Weak: claims PP=2 halves latency because each node does half the layers.

Hire: explains sequential traversal, states the bubble formula and shows that continuous decoding with $p$ micro-batches removes it.

Strong Hire: notes that each micro-batch rereads the weights in the bandwidth-bound regime, so PP needs about $p$ times the concurrency, and computes the inter-node all-reduce latency at which PP beats cross-node TP.

Common mistakes

• Applying the training bubble to steady-state decoding, where the pipeline need not drain.
• Ignoring that micro-batches are separate weight reads, then promising batch-128 throughput from two micro-batches of 64.
• Putting the PP boundary inside a node and TP across nodes, which sends the chatty traffic over the slow link.
• Forgetting stage imbalance: the LM head and embedding land on the end stages, and the slowest stage sets the period.