all to all

Every MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

Compare expert counts with rank-level completion times. Explain why the longest combine span can occur on a lightly loaded rank, then test a placement change against a communication control.