TL;DR:
gpu_memory_utilizationcaps vLLM's share of each GPU's total memory (0.92 by default in v0.31), and whatever is left after weights, the activation peak and CUDA graphs becomes the KV pool.max_model_lensets the longest request, and startup fails unless the pool holds one request of that length.max_num_seqscaps concurrent sequences and sizes runner buffers, but on an 80 GB H100 serving Llama 3.1 8B the 416K-token pool binds long before the default 1,024 sequences do.
How to approach it
Separate the three budgets before talking about any of them: a memory fraction, a per-request length and a per-step concurrency cap. Then walk startup in the order the worker runs it, because the interaction only makes sense as a sequence: snapshot, load weights, profile, subtract, check. Ask which model, precision and GPU count, since the same flags pass on one deployment and fail on the next.
A strong answer
Checked against the vLLM v0.31.0 source (October 2026):
- Snapshot. Each worker records free and total device memory, then computes its budget as
ceil(total x gpu_memory_utilization). The fraction is of total memory, not of what is free, and the worker refuses to start if free memory is already below the budget. Two instances on one GPU must therefore split the fraction (0.45 each), not both ask for 0.9. - Load and profile. Weights load, then a dummy forward pass at
max_num_batched_tokensmeasures the activation peak, and the runner estimates CUDA-graph memory (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHSdefaults to on, so graph memory now comes out of the KV budget instead of surprising you later). - Subtract. The KV pool is the budget minus everything non-KV. It is carved into blocks of 16 tokens.
- Check. If the pool cannot hold one request of
max_model_len, startup raises an error that names the needed and available GiB and an estimated maximum length.max_model_lendefaults to the model config's context length, 131,072 for Llama 3.x, so a default launch asks for the worst case.
Then the scheduler applies max_num_seqs every step as a cap on running requests. It is not a memory reservation; blocks are allocated as tokens arrive.
For Llama 3.1 8B in BF16 on one H100, with an assumed 3 GB for activations, graphs and workspace:
budget = 80 GB x 0.92 = 73.6 GB
weights = 8.03e9 params x 2 B = 16.06 GB
KV pool = 73.6 - 3.0 - 16.06 = 54.5 GB (50.8 GiB)
KV per token = 2 x 32 layers x 8 kv_heads x 128 x 2 B = 128 KiB
pool in tokens = 54.5e9 / 131,072 = 416,103 tokens
one 131K request = 131,072 x 128 KiB = 16 GiB -> check passes
concurrency at max length = 416,103 / 131,072 = 3.2 requests
sanity: at 4K context the pool holds 101 sequences; 1,024 would need 512 GiB
The startup log prints exactly these two numbers, the KV cache size in tokens and the maximum concurrency at max_model_len. They answer the capacity question better than any flag.
Now the failure. Llama 3.3 70B in BF16 on two H100s at TP=2:
budget, 2 GPUs = 2 x (73.6 - 3.0) = 141.2 GB
weights = 70.55e9 x 2 B = 141.1 GB
KV pool = 141.2 - 141.1 = 0.09 GB
needed at 131K = 131,072 x 320 KiB = 40 GiB -> startup error
The error suggests raising gpu_memory_utilization or lowering max_model_len. Neither helps here: at 0.95 the pool is 4.9 GB, about 15K tokens, which is a server that holds one long request. The real fix is FP8 weights (70.6 GB of headroom, 215K tokens, 1.6 requests at full length) or TP=4. Raising the fraction toward 1.0 also eats the slack that keeps fragmentation and NCCL buffers from turning into a runtime OOM.
How the three interact, in one rule each:
| Flag | What it bounds | When it binds first |
|---|---|---|
gpu_memory_utilization | KV pool size, through the budget | Large models, long contexts |
max_model_len | Longest single request; the startup check | Default 128K context on tight memory |
max_num_seqs | Running requests per step; graph and buffer sizing | Short contexts, small models |
Setting max_model_len=-1 asks vLLM to binary-search the largest length that fits, which turns the startup error into a smaller context window. That is right for a dev box and wrong for production, where the product decides the context, and memory follows.
The reversal: when contexts are short (chat at 1K tokens on an 8B model, where the pool holds over 400 sequences), max_num_seqs does bind, and raising it, or lowering it to protect TPOT, is the lever that moves.
What interviewers probe next
"Two vLLM instances on one GPU, both at 0.5. The second one fails." The first instance's CUDA context and allocator overhead sit outside its budget, so free memory is below 0.5 of total when the second starts. Set 0.45 each, or pin the KV size with kv_cache_memory_bytes, which skips the fraction entirely.
"TensorRT-LLM's free_gpu_memory_fraction is 0.9. Same thing?" No. In TensorRT-LLM v1.4.0rc0 the fraction multiplies the memory free after the model loads and goes to KV only, so 0.9 there and 0.92 in vLLM are different budgets.
"Why does max_num_seqs affect memory at all if it reserves nothing?" It sizes the runner's per-request buffers, and the default CUDA-graph ceiling is the smaller of twice max_num_seqs and 512 tokens on Hopper, so the value changes what gets captured and how much graph memory the profile subtracts.
How it's scored
Weak: describes gpu_memory_utilization as the KV fraction, or treats max_num_seqs as guaranteed concurrency.
Hire: walks snapshot, weights, profile, subtract, check; computes the pool in tokens and the concurrency at max_model_len; explains why the 70B TP=2 launch fails.
Strong Hire: rejects raising the fraction as the fix and picks FP8 weights or TP=4 with the numbers, knows graph memory is now in the profile, and contrasts the TensorRT-LLM fraction's different base.
Common mistakes
- Reading the fraction as "of free memory", then running two instances at 0.9 and 0.9.
- Leaving
max_model_lenat the 131,072 default for a product whose prompts never pass 8K, and paying for it in a failed check or a smaller batch. - Raising
max_num_seqsto fix low throughput when the log shows the pool holds only 3.2 requests at full length. - Quoting 0.9 as the default; that changed to 0.92 in v0.20.