cuda graphs

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

Every kernel in the step can run near peak bandwidth while the step itself runs at two thirds of it. The answer reads the three rows of the timeline that tell you why, and turns the gaps into milliseconds.

A graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.