load balancing
Inference interview questions tagged load balancing, across every topic.
3 questions · 1 unlocked for you
Concepts behind "load balancing"
Concept pages for the mechanisms these questions lean on.
Advanced
Expert Parallelism and the MoE All-to-AllExpert parallelism places different experts on different GPUs and moves tokens to them with two all-to-alls per MoE layer. This page counts those bytes for DeepSeek-V3, shows why EP exists (it gathers enough tokens per expert to leave the bandwidth-bound regime), and simulates the load imbalance that makes the slowest GPU set the step time.🕸️ Distributed Inference🔒 Premium
Advanced
Prefix-Aware and KV-Aware RoutingWhy a prefix cache is only as good as the router in front of it: round-robin spreads each shared prefix across every replica and shrinks the hit rate, prefix-affinity hashing concentrates it, and KV-aware routing scores replicas on cached blocks against load so one hot tenant cannot melt a node.🏛️ Inference System Design🔒 Premium