Practice tests · 30 items

Distributed & Disaggregated Inference practice test

Tensor, pipeline, expert and data-parallel attention, the all-reduce bytes per layer, MoE dispatch and combine all-to-all, prefill and decode disaggregation, KV transfer over NVLink and RDMA, and rack-scale NVLink domains. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test

Topic

Difficulty

Questions

10 in this pool · about 7 min

Reveal answers

Sample items with answers

easy · sample

In Megatron-style tensor parallelism, how many all-reduces does one transformer layer need in the forward pass?

One, at the end of the layer, summing attention and MLP partials together

Four: after QKV, after attention, after the up projection, after the down projection

Zero, because each GPU computes whole heads and never needs other GPUs' outputs

✓Two: after the attention output projection and after the MLP down projection

Megatron splits the first matrix of each block by columns (QKV by heads, gate and up by intermediate width) and the second by rows (output projection, down projection). The column-split output is exactly the sharded input the row-split matrix needs, so nothing moves between them, but the row-split matmul leaves every GPU with a partial sum of the full hidden state. Summing those is one all-reduce per block, two per layer: 160 per decode step for 80-layer Llama 3.3 70B. Heads stay local, but the output projection mixes them.

easy · sample

Why does pipeline parallelism not reduce per-token decode latency at batch 1, while tensor parallelism does?

Pipeline stages each hold a full copy of the weights, so every stage still reads the entire model per token

✓Each token passes through the stages one after another, so its latency stays the sum of every stage's time

Pipeline parallelism applies only to prefill, and decode falls back to the GPUs of the first stage alone

The activations sent between stages are larger than the weights, so the transfer dominates every step

A token must leave stage 1 before stage 2 can start on it, so with one sequence in flight only one stage works at a time and latency is the sum of the stage times. Each stage reads only its own layers, but serially. Tensor parallelism splits every layer across GPUs that run at the same moment, so each reads 1/p of the weights per step. The activation passed between stages is small: one 16,384-wide BF16 hidden state is 32 KiB for Llama 3.1 405B. PP buys capacity, and throughput once micro-batches fill it.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: