TL;DR: A CUDA graph replays a recorded kernel sequence with fixed shapes and pointers, so one graph serves exactly one batch size; engines capture a ladder of sizes (51 by default in vLLM v0.31 on an H100, up to 512 tokens) and pad each step up to the next rung. In bandwidth-bound decode the padding is close to free: padding 257 to 272 rows on Llama 3.1 8B adds 0.24 ms of compute inside a 4.79 ms linear-phase weight-read bound. The real costs are graph memory, capture time at startup and the batches that fall off the top of the ladder.
How to approach it
Start with what a graph removes: the CPU cost of launching a few hundred small kernels per decode step. Then say why that forces one graph per shape, and only then price the padding, by regime. Ask what the model size and decode batch are, because launch overhead matters most when the step is short.
A strong answer
Why graphs. A decode step for Llama 3.1 8B in BF16 on an H100 SXM reads 16.06 GB of weights:
memory floor = 16.06e9 B / 3.35e12 B/s = 4.8 ms
kernels/step ~ 32 layers x 12 kernels + ~10 others = ~394 launches
launch cost = 394 x 4 us (assumed CPU+driver cost) = 1.6 ms
If the GPU waits on the CPU between small kernels, a third of the step is overhead. A graph records the whole sequence once and replays it with one launch, so the GPU sees back-to-back kernels.
Why one per size. Capture freezes every kernel's grid dimensions, arguments and buffer addresses. A GEMM over 64 rows and one over 72 rows are different launches, and attention metadata is sized by batch. So the engine copies each step's inputs into static buffers and replays the graph recorded for a size at least as large, with the extra rows as padding.
The ladder in vLLM v0.31.0. With no explicit sizes, the ceiling is the smaller of twice max_num_seqs and a platform default of 512 tokens (1,024 on data-center Blackwell), clipped to max_num_batched_tokens. The sizes are [1, 2, 4], then steps of 8 up to 248, then steps of 16 up to the ceiling. V1 flattens the batch, so these are token counts; a plain decode step has one token per request. The default mode, FULL_AND_PIECEWISE, captures full graphs for decode batches and piecewise graphs (attention left outside) for mixed prefill steps.
ladder size, H100 = 3 + 31 (8..248) + 17 (256..512) = 51 sizes
ladder size, B200 = 3 + 31 + 49 (256..1024) = 83 sizes
worst padding = 9 -> 16: 7 of 16 rows wasted = 44%
257 -> 272: 15 of 272 = 5.5%
Pricing the padding. Evaluate the linear and attention phases separately. Padding may leave real KV-history reads unchanged if the attention backend skips dummy rows, but it can still add activation traffic, metadata work and different kernel shapes. Do not assume it adds zero bytes.
The following analytical approximation uses 16.06 GB of streamed BF16 weights, 8.03 billion multiplied parameters, H100 SXM dense BF16 at 989 TFLOP/s and 3.35 TB/s. At 1,024 cached tokens per real request:
linear weight-read bound = 16.06e9 / 3.35e12 = 4.794 ms
linear compute, 257 rows = 2 x 8.03e9 x 257 / 989e12 = 4.173 ms
linear compute, 272 rows = 2 x 8.03e9 x 272 / 989e12 = 4.417 ms
attention KV-read bound = 257 x 1,024 x 131,072 / 3.35e12 = 10.297 ms
sequential bound = max(4.794, 4.417) + 10.297 = 15.091 ms
extra linear compute = 15 x 16.06e9 / 989e12 = 0.244 ms
sanity: both linear shapes stay below their own weight-read bound;
no overlap with the later attention phase is needed for this argument
This predicts little extra linear time in the idealized regime, not zero measured overhead. Above the linear ridge, padded rows increase the compute bound even when the later attention phase has much larger memory cost. Compare actual padded and unpadded kernels, and check the resident KV budget before testing large batches.
Where the cost actually is.
| Cost | Why | What to do |
|---|---|---|
| Graph memory | Captures retain workspace; compatible pools can share allocations | Check whether graph estimation is deducted from the KV budget |
| Startup time | Capture runs every size for each mode | Fewer sizes on autoscaled fleets |
| Off the ladder | Batches above the ceiling run without full graphs | Raise max_cudagraph_capture_size |
The third row is the trap. With the default max_num_seqs of 1,024 on an H100, decode batches between 513 and 1,024 run without full graphs. On a small model at very high concurrency that is where throughput falls off, and the fix is a larger ceiling, not more padding rungs.
Memory accounting needs a configuration check. In the vLLM v0.31.0 GPU worker, subtracting the graph-memory estimate depends on VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS; an explicit KV-memory setting follows a different path. Compare the estimate with actual post-capture memory and available KV pages. More captures can shrink a profiled pool or consume remaining headroom, depending on that path (checked October 18, 2026).
Prefill is the opposite regime: an 8,192-token chunk on the same model is about 133 ms of compute, so 1.6 ms of launch overhead is 1% and graphs barely matter. That is why mixed steps get piecewise graphs and large prefills run without them.
The reversal condition: for a 405B model at TP=8 a decode step is tens of milliseconds and launch overhead is a few percent, so a short ladder, or even eager mode for debugging, costs little. Graphs pay most for small models at small batch.
What interviewers probe next
"How does TensorRT-LLM do it?" In v1.4.0rc0 the PyTorch backend's CudaGraphConfig defaults to a maximum batch of 128 with enable_padding off, so a batch with no exact match runs eagerly. Its docstring warns each graph can use up to 200 MB.
"Why not capture every size from 1 to 1,024?" A thousand captures multiply startup time and graph memory for padding that was already nearly free.
"Speculative decoding sends 4 tokens per request." Uniform decode steps are then multiples of 4 tokens, so vLLM v0.31 adds request-count sizes scaled by the query length; a token-strided rung that is not a multiple would never be used.
How it's scored
Weak: says graphs make kernels faster, or that padding wastes the padded fraction of the step.
Hire: explains frozen shapes and static buffers, quantifies launch overhead against the step time, and shows padding is hidden in the bandwidth-bound regime.
Strong Hire: names the default ladder and its ceiling, spots batches above the ceiling falling off graphs, and moves the cost discussion to memory and startup time.
Common mistakes
- Charging padding at the padded fraction (44% at batch 9) instead of at FLOPs against a memory-bound step.
- Assuming
max_num_seqs=1024means graphs up to 1,024; the default ceiling on Hopper is 512. - Benchmarking with
enforce_eageron to save startup time and reporting the result as the engine's decode speed. - Forgetting that graph replay needs static input addresses, then writing a custom sampler that allocates per step.