Advanced

Preemption: Swap Versus Recompute

When the KV pool runs out mid-decode, the scheduler must evict a running sequence: either swap its cache to host memory over PCIe and copy it back later, or free it and recompute it with a fresh prefill. The cost of each depends on model size, KV bytes per token and whether prefix caching still holds the freed blocks.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Iteration-Level Scheduling (Continuous Batching)Chunked Prefill and the Token Budget Paged KV Memory and Block Tables The KV Cache: Bytes per Token and the Memory Budget Prefix Caching and Chained Block Hashes Autoscaling LLM Fleets on the Right Signal

PRACTICE THIS IN REAL QUESTIONS

KV Cache & GPU MemoryDerive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.→Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Implement-from-Scratch CodingImplement a paged KV-cache block manager with copy-on-write forking, then a scheduler step that preempts when blocks run out.→KV Cache & GPU MemoryWhy does a paged KV cache waste at most one block per sequence, and how would you pick the block size?→Attention KernelsWrite the causal mask for a decode step with a KV cache, where the query is shorter than the keys. Which corner does it align to?→