One replica in eight is 30% slower than the rest. Find it from the metrics and explain the likely hardware causes.
Same model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
Same model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.
At batch 1 the step reads every weight once to produce one token per sequence, so arithmetic intensity sits near 1 FLOP per byte and the tensor cores idle.