Advanced

Prefix Caching and Chained Block Hashes

Automatic prefix caching keeps the KV blocks of finished requests and reuses them when a new prompt starts with the same tokens. Each full block is keyed by a hash chained from its parent block, which makes a hit mean the whole prefix matched; that decides TTFT for shared system prompts and how much cache multi-turn traffic needs.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Paged KV Memory and Block Tables The KV Cache: Bytes per Token and the Memory Budget SGLang's Radix Cache and Overlap Scheduler Prefix-Aware and KV-Aware Routing Prefill Versus Decode: Two Workloads on One GPU Chunked Prefill and the Token Budget

LESSONS THAT TEACH THIS

Serving under pressure: from admission to releaseReserve the page a request has not written yet12 min→Serving under pressure: from admission to releaseCancel the request without reusing a live page12 min→

PRACTICE THIS IN REAL QUESTIONS

KV Cache & GPU MemoryDerive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.→Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→KV Cache & GPU MemoryWhy does a paged KV cache waste at most one block per sequence, and how would you pick the block size?→