← ⚙️ Serving Engines
Advanced
SGLang's Radix Cache and Overlap Scheduler
SGLang stores every cached prefix in a radix tree keyed by token ids, so a hit can end at any token rather than at a block boundary, and its overlap scheduler prepares the next batch while the GPU runs the current one. This page walks both through the v0.5.21 source: matching, lock references, leaf-first LRU eviction, the 128-request cutoff on prefix-aware scheduling, and memory defaults.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→KV Cache & GPU MemoryWhy does a paged KV cache waste at most one block per sequence, and how would you pick the block size?→Inference System DesignDesign prefix-aware routing that keeps cache hits high without overloading the replica that owns a hot prefix. What is your fallback rule?→Serving Engine InternalsWalk through how vLLM's prefix caching identifies a reusable KV block and hands it to a new request, step by step.→