Advanced

Expert Parallelism and the MoE All-to-All

Expert parallelism places different experts on different GPUs and moves tokens to them with two all-to-alls per MoE layer. This page counts those bytes for DeepSeek-V3, shows why EP exists (it gathers enough tokens per expert to leave the bandwidth-bound regime), and simulates the load imbalance that makes the slowest GPU set the step time.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Tensor Parallelism for Serving Prefill and Decode Disaggregation The Roofline Model for LLM Inference Weight-Only Quantization Versus W8A8 Capacity Planning in Tokens per Second

PRACTICE THIS IN REAL QUESTIONS

Inference System DesignDesign prefix-aware routing that keeps cache hits high without overloading the replica that owns a hot prefix. What is your fallback rule?→Distributed & Disaggregated InferenceSize the expert-parallel all-to-all for a 256-expert, top-8 MoE at batch 256. How many bytes per token, and how long on the network?→Distributed & Disaggregated InferenceWhy do MLA models like DeepSeek V3 run data-parallel attention with expert-parallel MoE instead of plain tensor parallelism?→Inference Physics & RooflineDeepSeek V3 has 671B parameters but only 37B active. Why does a decode step at batch 64 read far more than 37B?→Inference System DesignDesign a multi-region deployment for a 70B model that survives the loss of a whole region. How much spare capacity do you buy, and where?→Distributed & Disaggregated InferenceSome experts get far more tokens than others. What do hot experts do to step time, and how do redundant experts and rebalancing fix it?→