Advanced

Chunked Prefill and the Token Budget

Chunked prefill splits a long prompt into pieces that ride along with decode tokens, under a per-step token budget such as vLLM's max_num_batched_tokens. The budget controls how much prompt work shares a step with decode, trading prompt progress against token gaps without guaranteeing a fixed latency.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

Iteration-Level Scheduling (Continuous Batching)Prefill Versus Decode: Two Workloads on One GPU The Roofline Model for LLM Inference Preemption: Swap Versus Recompute Prefill and Decode Disaggregation Serving Engine Knobs and Their Defaults

LESSONS THAT TEACH THIS

Serving under pressure: from admission to releaseChoose a prefill chunk from the next-token deadline12 min→

PRACTICE THIS IN REAL QUESTIONS

Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Inference System DesignDesign the request path for an LLM API: gateway, router, engines, streaming. Where is each SLO measured, and what does each hop cost?→Performance Debugging & BenchmarkingTTFT is fine but inter-token latency is bad. Decompose the decode step and rank the causes.→Inference Physics & RooflineWhy does prefill turn compute-bound after a few hundred tokens on an H100, and how long does a 4K-token prompt take on a 70B model?→