04Size the expert-parallel all-to-all for a 256-expert, top-8 MoE at batch 256. How many bytes per token, and how long on the network?▼hardNewNVIDIAGoogle DeepMindFireworks AIunlockedEvery MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.Open full answer →
05Why do MLA models like DeepSeek V3 run data-parallel attention with expert-parallel MoE instead of plain tensor parallelism?▼expertNewNVIDIAFireworks AITogether AIunlockedTensor parallelism splits a GQA cache by KV head. MLA has no KV heads to split, so TP copies the whole cache onto every GPU without any warning. The answer measures that waste on an H200 node and prices the fix.Open full answer →
09Some experts get far more tokens than others. What do hot experts do to step time, and how do redundant experts and rebalancing fix it?▼expertNewNVIDIAGoogle DeepMindMistral AI○ sign inExpert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.Open full answer →