Google Distributed & Disaggregated Inference interview questions

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

A 1 TB FP8 model fits on one B200 node and needs four H100 nodes. That single fact sorts the two generations into roles. The answer assigns them, sizes both pools, and then deals with the precision mismatch that shows up when the two halves disagree about the weights.

Tensor parallelism divides the weight bytes but not the all-reduce count, so each doubling buys less than the one before. The answer prices that fixed cost, then finds the workload where TP=8 wins anyway.

Two matrix multiplies in a row, and the order of the splits decides whether you communicate once or twice per block. The answer places every split for a 70B layer and prices what the remaining all-reduces send per token.

A token still has to pass through every layer in order, so splitting the layers across nodes cannot make one token faster. The answer shows what pipelining buys instead, prices the bubble, and finds the all-reduce latency at which it beats cross-node tensor parallelism.

Tensor parallelism splits a GQA cache by KV head. MLA has no KV heads to split, so TP copies the whole cache onto every GPU without any warning. The answer measures that waste on an H200 node and prices the fix.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

Same eight GPUs, same weights per GPU, same KV capacity. One layout pays 252 all-reduces per token, the other pays for every layer twice in wall-clock time. The answer works both, then finds the precision choice that forces the second.

Eight GPUs per NVLink domain shaped every serving layout of the Hopper era. Seventy-two moves the boundary that expert parallelism and KV transfer used to cross over the NIC. The answer prices what moves inside the rack, what still does not fit, and the failure mode that comes with it.

Collectives run at the speed of their slowest link, so one NIC that trained at a quarter of its rate slows every GPU in the instance by the same amount. The answer shows the arithmetic that makes a 4x slower link a 2x slower step, then bisects the cluster to the culprit.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

Pulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.