04Size the expert-parallel all-to-all for a 256-expert, top-8 MoE at batch 256. How many bytes per token, and how long on the network?▼hardNewNVIDIAGoogle DeepMindFireworks AIunlockedEvery MoE layer sends each token out to its experts and gathers the results back, twice per layer, 58 times per step. The answer turns DeepSeek V3's config into bytes per token, splits them between NVLink and the NIC, and shows what node-aware dispatch saves.Open full answer →
06You split prefill and decode onto separate nodes. Compute the KV transfer per request and the network bandwidth it needs.▼hardNewNVIDIAPerplexityFireworks AI○ sign inAn 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.Open full answer →
14You need to get 500 GB of weights onto 1,000 GPUs as fast as possible. What is the lower bound, and what design gets close to it?▼mediumNewOpenAIAnthropicMeta◆ premiumPulling every replica's copy from object storage turns a 500 GB model into 62.5 TB of reads. The answer finds the three bounds that actually limit a fleet load, shows a pipelined broadcast reaching the tightest one, and lists the startup work that hides behind the bytes.Open full answer →