system design
Inference interview questions tagged system design, across every topic.
2 questions · 0 unlocked for you
Concepts behind "system design"
Concept pages for the mechanisms these questions lean on.
Foundational
Capacity Planning in Tokens per SecondHow to size an LLM serving fleet: convert requests into prompt and output tokens per second, price prefill by FLOPs and decode by bytes, cap utilization for tail latency, check KV residency and TPOT, then turn replicas into a cost per million tokens.🏛️ Inference System Design
Advanced
Prefix-Aware and KV-Aware RoutingWhy a prefix cache is only as good as the router in front of it: round-robin spreads each shared prefix across every replica and shrinks the hit rate, prefix-affinity hashing concentrates it, and KV-aware routing scores replicas on cached blocks against load so one hot tenant cannot melt a node.🏛️ Inference System Design🔒 Premium
Advanced
Autoscaling LLM Fleets on the Right SignalWhy GPU utilization and requests per second are the wrong autoscaling signals for LLM serving, which engine metrics lead and which lag, how to turn token rates into a replica target, and why a five-minute cold start sets the headroom you must carry.🏛️ Inference System Design🔒 Premium