← 🏛️ Inference System Design
Advanced
Autoscaling LLM Fleets on the Right Signal
Why GPU utilization and requests per second are the wrong autoscaling signals for LLM serving, which engine metrics lead and which lag, how to turn token rates into a replica target, and why a five-minute cold start sets the headroom you must carry.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Distributed & Disaggregated InferenceGoing from TP=4 to TP=8 on one H100 node cut a 70B model's latency by less than half and lowered tokens per GPU. One TP=8 or two TP=4?→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Performance Debugging & BenchmarkingTTFT is fine but inter-token latency is bad. Decompose the decode step and rank the causes.→Batching, Scheduling & AdmissionUse Little's law on an LLM server: how many requests are in flight at 10 per second, and does the KV cache hold them?→Inference System DesignDesign prefix-aware routing that keeps cache hits high without overloading the replica that owns a hot prefix. What is your fallback rule?→