← 🟩 GPU Performance
Advanced
CUDA Graphs and Launch Overhead in Decode
A decode step is hundreds of small kernels, and once tensor parallelism or a small model shrinks the GPU time per step, the CPU cost of launching them becomes the step time. CUDA graphs record the whole step once and replay it with one launch, which is why every serving engine captures decode graphs per batch-size bucket.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→Batching, Scheduling & AdmissionQuantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.→Serving Engine InternalsWalk one request through vLLM V1, from the HTTP call to the streamed token. Which process owns each step, and why is it split that way?→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Performance Debugging & BenchmarkingAfter a prompt-template deploy, p99 TTFT went from about 100 ms to over a second while GPU utilization did not move. Debug it.→Inference Physics & RooflineDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?→