Advanced

Prefix-Aware and KV-Aware Routing

Why a prefix cache is only as good as the router in front of it: round-robin spreads each shared prefix across every replica and shrinks the hit rate, prefix-affinity hashing concentrates it, and KV-aware routing scores replicas on cached blocks against load so one hot tenant cannot melt a node.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Prefix Caching and Chained Block Hashes Capacity Planning in Tokens per Second Autoscaling LLM Fleets on the Right Signal SGLang's Radix Cache and Overlap Scheduler Prefill and Decode Disaggregation Latency Decomposition: Queue, Prefill, Decode

PRACTICE THIS IN REAL QUESTIONS

KV Cache & GPU MemoryDerive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→KV Cache & GPU MemoryWhy does a paged KV cache waste at most one block per sequence, and how would you pick the block size?→Attention KernelsWrite the causal mask for a decode step with a KV cache, where the query is shorter than the keys. Which corner does it align to?→Quantization & NumericsFP8 comes in E4M3 and E5M2. Give the range and precision of each, and say which tensors use which in inference.→