07One replica in eight is 30% slower than the rest. Find it from the metrics and explain the likely hardware causes.▼mediumNewNVIDIATogether AIBaseten○ sign inSame model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.Open full answer →
10Implement a ring all-reduce as a simulation over p ranks, verify it against a direct sum, and count the bytes each rank sends.▼mediumNewNVIDIAGoogle DeepMindMeta○ sign inTwo loops and an index formula, and most attempts get the index wrong by one. The answer writes reduce-scatter and all-gather over a simulated ring, proves the result matches a direct sum when p does not divide the length, and turns the byte count into the reason small messages avoid the ring.Open full answer →