Skip to content
← Distributed & Disaggregated Inference / 09
expertNewNVIDIAGoogle DeepMindMistral AI

Some experts get far more tokens than others. What do hot experts do to step time, and how do redundant experts and rebalancing fix it?

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackA GB200 NVL72 puts 72 GPUs in one NVLink domain. What does that change for MoE serving and for prefill-decode disaggregation?Next in this trackA 128k-token prefill takes 10 seconds on one node. How does context parallelism split it across nodes, and what does it send?Next in this trackA multi-node MoE deployment's TPOT doubled overnight, and one network link is degraded. How do you find it, and how do you design around it?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.