07Why does a one-shot custom all-reduce beat NCCL for small decode messages on an NVLink node, and when does it lose?▼hardNewNVIDIAAMDFireworks AI○ sign inA ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.Open full answer →
12A multi-node MoE deployment's TPOT doubled overnight, and one network link is degraded. How do you find it, and how do you design around it?▼mediumNewNVIDIAOpenAIMeta◆ premiumCollectives run at the speed of their slowest link, so one NIC that trained at a quarter of its rate slows every GPU in the instance by the same amount. The answer shows the arithmetic that makes a 4x slower link a 2x slower step, then bisects the cluster to the culprit.Open full answer →
10Implement a ring all-reduce as a simulation over p ranks, verify it against a direct sum, and count the bytes each rank sends.▼mediumNewNVIDIAGoogle DeepMindMeta○ sign inTwo loops and an index formula, and most attempts get the index wrong by one. The answer writes reduce-scatter and all-gather over a simulated ring, proves the result matches a direct sum when p does not divide the length, and turns the byte count into the reason small messages avoid the ring.Open full answer →