THE INFERENCE CURRICULUM

The mechanisms behind every inference question

One page per mechanism: the KV cache budget, the roofline, paged allocation, online softmax and the rest. Each states the governing equation, works one example with real part numbers, and lists the questions that depend on it. Read the tracks in order, or open the one your last mock exposed.

36 concepts across 12 tracks · foundational concepts are free · see them as a map

Begin the curriculum →

⚛️ Inference Physics

Count bytes and FLOPs per token, place prefill and decode on the roofline, and turn a model config and a datasheet into a latency floor before any benchmark runs.

START HERE

01The Roofline Model for LLM InferenceFree

The roofline model bounds a kernel's speed by the smaller of peak compute and bandwidth times arithmetic intensity. For LLM serving it decides whether a step is priced in bytes or in FLOPs: batch-1 decode sits near 1 FLOP/byte against an H100 ridge of 295, prefill sits far above it.

02Prefill Versus Decode: Two Workloads on One GPUPremium

Every LLM request runs two different workloads: prefill processes the whole prompt in one compute-bound pass, then decode generates one token per step and is bound by weight and KV reads. Which one dominates a request decides the metric you tune, the batching policy and whether the two phases belong on the same GPU.

03Model Bandwidth Utilization (MBU)Premium

MBU divides the bytes a decode step must read (weights plus KV cache) by the measured step time and the chip's peak bandwidth. It is the decode-side twin of MFU, and it decides whether a slow server has a software problem to fix or has already hit the hardware floor.

🧮 KV Cache & Memory

Derive the KV footprint for every attention variant, then manage it like an operating system: pages, sharing, eviction, quantization and tiers.

01The KV Cache: Bytes per Token and the Memory BudgetFree

The KV cache stores every past token's keys and values so decode never recomputes them. Its size per token is fixed by the attention design, and the memory left after weights and the engine's reserve decides how many sequences fit, which is the number that sets batch size, concurrency and when preemption starts.

02Paged KV Memory and Block TablesPremium

Paged KV memory splits the cache into fixed-size blocks and gives each sequence a block table that maps logical positions to physical blocks, as virtual memory does with pages. It bounds waste to one partial block per sequence, which decides how many sequences fit and makes prefix sharing and copy-on-write possible.

03Prefix Caching and Chained Block HashesPremium

Automatic prefix caching keeps the KV blocks of finished requests and reuses them when a new prompt starts with the same tokens. Each full block is keyed by a hash chained from its parent block, which makes a hit mean the whole prefix matched; that decides TTFT for shared system prompts and how much cache multi-turn traffic needs.

⏱️ Batching & Scheduling

Why iteration-level scheduling won, how a token budget trades TTFT against inter-token latency, and what a scheduler does when memory runs out.

01Iteration-Level Scheduling (Continuous Batching)Free

Iteration-level scheduling, sold as continuous or in-flight batching, re-forms the batch at every forward step instead of once per batch of requests. Finished sequences leave and queued ones join on the next step, which decides how much of a GPU's decode capacity real traffic with uneven output lengths can use.

02Chunked Prefill and the Token BudgetPremium

Chunked prefill splits a long prompt into pieces that ride along with decode tokens, under a per-step token budget such as vLLM's max_num_batched_tokens. The budget controls how much prompt work shares a step with decode, trading prompt progress against token gaps without guaranteeing a fixed latency.

03Preemption: Swap Versus RecomputePremium

When the KV pool runs out mid-decode, the scheduler must evict a running sequence: either swap its cache to host memory over PCIe and copy it back later, or free it and recompute it with a fresh prefill. The cost of each depends on model size, KV bytes per token and whether prefix caching still holds the freed blocks.

🔦 Attention Kernels

Exact attention computed in tiles: the online-softmax recurrence, the FlashAttention generations, and the decode-time kernels that split work across the KV length.

01The Online Softmax RecurrenceFree

Online softmax keeps a running max, a running normalizer and an unnormalized output, and rescales them when the max grows. The merge rule is exact and associative, which is what lets FlashAttention tile over keys and Flash-Decoding split one long context across thread blocks.

02FlashAttention 1 to 4: What Each Generation ChangedPremium

Every FlashAttention generation computes the same exact attention and attacks a different bottleneck: HBM traffic in FA1, poor work partitioning in FA2, Hopper's asynchronous tensor cores in FA3, and Blackwell's slow exponential units in FA4. Knowing which bottleneck each one removed tells you which kernel a given GPU should run.

03Split-KV Decoding (Flash-Decoding)Premium

In decode there is one query row per sequence, so a FlashAttention kernel parallelized over batch and heads launches too few thread blocks to pull full HBM bandwidth. Split-KV divides each sequence's keys across blocks and merges the partial results exactly with their log-sum-exp, which decides whether long-context decode reaches bandwidth.

🟩 GPU Performance

The hardware model an inference kernel engineer reasons with: hierarchy bandwidths, tensor-core pipelines per generation, and the profiler evidence that separates hypotheses.

01The GPU Memory Hierarchy in NumbersFree

An H100 SXM has 33 MiB of registers, up to 29 MiB of shared memory, a 50 MB L2 and 80 GB of HBM3 at 3.35 TB/s. Knowing those capacities decides which inference tensors can be reused on chip, why decode streams every weight from HBM each step, and when a benchmark is secretly measuring L2.

02Bytes in Flight: Little's Law for GPU MemoryPremium

Little's law says sustained memory bandwidth equals the bytes a kernel keeps outstanding divided by memory latency. On an H100 SXM that is about 2 MB across the GPU, or 15 KB per SM, and it decides whether a bandwidth-bound decode kernel can reach peak at all, before any question of coalescing or caching.

03CUDA Graphs and Launch Overhead in DecodePremium

A decode step is hundreds of small kernels, and once tensor parallelism or a small model shrinks the GPU time per step, the CPU cost of launching them becomes the step time. CUDA graphs record the whole step once and replay it with one launch, which is why every serving engine captures decode graphs per batch-size bucket.

🔢 Quantization & Numerics

What each number format can represent, where the scales live, which kernels make a format fast, and how to prove a quantized model still answers the same way.

01Number Formats for Inference: BF16, FP8, INT4, FP4Free

Inference formats differ in three ways that matter: bytes per element, which sets decode speed and memory; exponent bits versus mantissa bits, which sets range and precision; and whether the GPU has tensor cores that multiply them natively. Llama 3.3 70B is 141 GB in BF16, 71 GB in FP8 and about 40 GB in NVFP4.

02Weight-Only Quantization Versus W8A8Premium

Weight-only quantization (W4A16, W8A16) stores small weights but multiplies in BF16, so it speeds up bandwidth-bound decode and does nothing for prefill. W8A8 quantizes activations too and runs on FP8 or INT8 tensor cores. The crossover batch is set by the GPU, about 141 on a B200, and activation outliers decide how hard W8A8 is.

03Block-Scaled FP4: MXFP4 and NVFP4Premium

MXFP4 and NVFP4 both store E2M1 4-bit values with a shared scale per block. MXFP4 uses one power-of-two E8M0 scale per 32 elements (4.25 bits); NVFP4 uses one E4M3 scale per 16 plus an FP32 per-tensor scale (4.5 bits). The scale format decides how much of FP4's tiny grid each block actually uses.