nccl

A hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.

Two loops and an index formula, and most attempts get the index wrong by one. The answer writes reduce-scatter and all-gather over a simulated ring, proves the result matches a direct sum when p does not divide the length, and turns the byte count into the reason small messages avoid the ring.