Advanced

Prefill and Decode Disaggregation

Disaggregated serving runs prefill and decode on separate GPU pools and ships each request's KV cache between them. This page prices the interference it removes, the transfer it adds (2.5 GiB for an 8K-token Llama 3.3 70B prompt), how to size the prefill-to-decode ratio, and when the extra hop is not worth it.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Prefill Versus Decode: Two Workloads on One GPU Chunked Prefill and the Token Budget Prefix-Aware and KV-Aware Routing The KV Cache: Bytes per Token and the Memory Budget Tensor Parallelism for Serving

LESSONS THAT TEACH THIS

Serving under pressure: from admission to releaseChoose a prefill chunk from the next-token deadline12 min→

PRACTICE THIS IN REAL QUESTIONS

Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Inference Physics & RooflineDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?→Attention KernelsWrite the causal mask for a decode step with a KV cache, where the query is shorter than the keys. Which corner does it align to?→