Advanced

Prefill Versus Decode: Two Workloads on One GPU

Every LLM request runs two different workloads: prefill processes the whole prompt in one compute-bound pass, then decode generates one token per step and is bound by weight and KV reads. Which one dominates a request decides the metric you tune, the batching policy and whether the two phases belong on the same GPU.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

The Roofline Model for LLM Inference Model Bandwidth Utilization (MBU)Chunked Prefill and the Token Budget Prefill and Decode Disaggregation Latency Decomposition: Queue, Prefill, Decode The KV Cache: Bytes per Token and the Memory Budget

PRACTICE THIS IN REAL QUESTIONS

Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Quantization & NumericsAWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Inference Physics & RooflineDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?→