Going from TP=4 to TP=8 on one H100 node cut a 70B model's latency by less than half and lowered tokens per GPU. One TP=8 or two TP=4?

Question

InferenceInterviews · Accepted Answer

TL;DR: Each decode step does 160 all-reduces for an 80-layer model (two per layer, Megatron-style), and their latency does not shrink with the TP degree. With an assumed 10 µs per all-reduce, TP=8 brings a batch-1 step from 12.1 ms to 6.9 ms (1.76x, not 2x), and two TP=4 replicas on the same eight GPUs serve 12% to 19% more tokens per GPU. TP=8 still wins on latency, and on KV-bound long-context traffic: it stores the weights once, which leaves 448 GB for KV against 306 GB across two TP=4 replicas. How to approach it State the cost model before the conclusion: a decode step is weight streaming, which TP divides, plus communication, which TP does not divide and partly increases. Ask the latency target, the batch size the service runs at, and the context lengths, because the answer is "replicas for throughput, wider TP for latency or KV capacity" and the inputs decide which constraint binds. Ask what all-reduce implementation the engine uses, since that sets the fixed cost. A strong answer Llama 3.3 70B has 80 layers and hidden size 8,192. Each layer all-reduces the hidden state twice per token (after attention and after the MLP), so a decode step with batch $$B$$ performs 160 all-reduces of $$8192 	imes 2 	imes B$$ bytes. A ring all-reduce sends $$2(p-1)/p$$ of the buffer per GPU, and NVLink on an H100 moves 450 GB/s per direction (900 GB/s is the bidirectional figure). For small decode messages, latency dominates bandwidth, so the model is: $$ t{	ext{step}} = \underbrace{\frac{N b / p}{3.35\ 	ext{TB/s}}}{	ext{weights, divided by TP}} + 160\left(\alpha + \frac{2(p-1)}{p}\cdot\frac{16{,}384\,B}{450\ 	ext{GB/s}}ight) $$ $$\alpha$$, the fixed cost per all-reduce, is the number to measure, not to assume. Here it is taken as 10 µs, roughly what a general-purpose library pays for tiny messages, and 3 µs, roughly what a one-shot custom all-reduce over NVLink achieves: Communication grows from 13% of a TP=4 step to 23% of a TP=8 step at batch 1, and to 36% at batch 128 with the slower all-reduce. Two second-order costs push the same direction and this model leaves out: per-GPU matrices at TP=8 are small enough that GEMV kernels underfill the SMs and launch gaps grow relative to transfer time, and every GPU in the group waits for the slowest one at each all-reduce, so jitter compounds. So for throughput per GPU at a fixed batch, two TP=4 replicas win. The memory budget can reverse that. With vLLM's 0.92 memory fraction: TP=8 stores the weights once instead of twice, so it holds 46% more KV. For long-context traffic, where the KV pool rather than bandwidth limits the batch, the single TP=8 replica runs larger batches and can come out ahead on throughput too. The decision: two TP=4 replicas for short-context, throughput-priced traffic; TP=8 when the per-token latency target needs the lower step time, or when long contexts make KV capacity the binding limit. Make the choice per pool, and measure $$\alpha$$ on the actual node first, because it moves the answer more than any other input. What interviewers probe next "Why not TP=16 across two nodes?" The all-reduce then crosses the network: tens of microseconds per call, 160 times per token, on a 2.6 ms weight floor. Use pipeline or data parallelism across nodes and keep TP inside the NVLink domain. "Would FP8 weights change the choice?" Halving weight bytes halves the part TP divides and leaves the communication alone, so communication becomes a larger share and the case for narrower TP strengthens. "Overlap the communication?" Within a decode step the next layer needs the reduced result, so naive overlap does not help. Splitting the batch into two micro-batches and interleaving their compute and communication does, at the cost of smaller GEMMs. How it's scored Weak: expects TP=8 to be exactly twice as fast as TP=4, or proposes TP=16 across nodes for latency. Hire: counts two all-reduces per layer, separates the latency term from the bandwidth term, and explains why replicas win on throughput per GPU. Strong Hire: parameterizes the all-reduce cost and says how to measure it, quantifies the per-GPU throughput loss, and finds the KV-capacity reversal where TP=8 wins long-context throughput. Common mistakes Using 900 GB/s as one-direction NVLink bandwidth, which halves the bandwidth term. Treating the all-reduce payload as large: at batch 1 it is 16 KB, so it is latency-bound and bandwidth figures do not apply. Comparing one TP=8 replica with one TP=4 replica instead of with two, which is the real alternative on the same eight GPUs. Ignoring that KV heads (8 for this model) stop dividing beyond TP=8, so going wider replicates the cache.