Advanced

Model Bandwidth Utilization (MBU)

MBU divides the bytes a decode step must read (weights plus KV cache) by the measured step time and the chip's peak bandwidth. It is the decode-side twin of MFU, and it decides whether a slow server has a software problem to fix or has already hit the hardware floor.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

The Roofline Model for LLM Inference Prefill Versus Decode: Two Workloads on One GPU Inference Benchmarks That Do Not Lie CUDA Graphs and Launch Overhead in Decode Bytes in Flight: Little's Law for GPU Memory Weight-Only Quantization Versus W8A8

LESSONS THAT TEACH THIS

Serving under pressure: from admission to releaseQualify completed requests before accepting a speedup12 min→

PRACTICE THIS IN REAL QUESTIONS

Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→CUDA, Triton & GPU PerformanceYour decode GEMV kernel reaches only about half of HBM bandwidth on an H100. Find the reasons and get it above 80%.→Quantization & NumericsAWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Inference Physics & RooflineDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?→Attention KernelsWrite the causal mask for a decode step with a KV cache, where the query is shorter than the keys. Which corner does it align to?→