TL;DR: Start from the budget, not the device: vLLM may use
gpu_memory_utilization(0.92 by default in v0.31) of the GPU, and the KV pool is whatever is left of that after weights, the profiled activation peak and memory outside PyTorch. For Llama 3.3 70B in FP8 at TP=2 on H100s that is 73.6 GB minus 35.3 GB of weights minus about 3 GB of overheads, roughly 35.3 GB of KV per GPU, 215K tokens.
How to approach it
Draw the GPU as a stack and fill it bottom-up in the order the engine does at startup, because the KV pool is sized last and absorbs every error in the layers below it. Ask the model, weight precision, TP degree and the scheduler's token budget (max_num_batched_tokens, max_num_seqs), since the activation peak scales with the latter two. Then say which layers you would read from the startup log rather than estimate.
A strong answer
The worked case is Llama 3.3 70B with FP8 weights, BF16 KV cache, TP=2 on two H100 SXM 80 GB, an 8,192-token chunked-prefill budget and up to 256 running sequences. Figures marked "assumed" are typical magnitudes to replace with the log line on your own box.
| Layer | Per GPU | Where it comes from |
|---|---|---|
| Budget | 73.6 GB | 80 GB x 0.92 |
| CUDA context, cuBLAS workspaces | 0.6 GB, assumed | Outside PyTorch's allocator |
| Weights | 35.3 GB | 70.55e9 params x 1 B / 2 |
| Activation peak | 1.5 GB, estimated below | Profiling forward pass |
| CUDA graphs | 0.6 GB, assumed | Captured decode batch sizes |
| NCCL and custom all-reduce buffers | 0.3 GB, assumed | Communicators, IPC buffers |
| KV pool | 35.3 GB | Remainder |
| Not used by vLLM | 6.4 GB | The 8% outside the budget |
weights per GPU = 70,553,706,496 x 1 B / 2 = 35.28 GB (32.85 GiB)
activation peak, one 8,192-token chunk, TP=2:
gate+up out = 8,192 x (2 x 14,336) x 2 B = 0.47 GB
SiLU x up = 8,192 x 14,336 x 2 B = 0.23 GB
2 hidden bufs = 2 x 8,192 x 8,192 x 2 B = 0.27 GB
logits, 256 x 128,256 x 4 B, ~4 sampler copies = 0.53 GB
sum ~ 1.5 GB
KV per GPU = 73.6 - 0.6 - 35.28 - 1.5 - 0.6 - 0.3 = 35.32 GB
KV per token = 327,680 B / 2 (4 of 8 KV heads per GPU) = 163,840 B
tokens = 35.32e9 / 163,840 = 215,595
sanity: 215,595 / 8,192 = 26 full-length sequences; activations are 2% of the GPU
The order matters for what is measured and what is assumed. vLLM loads the weights and reports their size, then runs a profiling forward pass at the largest batch the scheduler can form and records the peak PyTorch allocation, then compares total used memory against what PyTorch accounts for, which catches the CUDA context and NCCL's own allocations as "non-torch" memory. The KV pool gets the rest, and the startup log prints both the available KV memory and the resulting token capacity. Read those two lines before any capacity argument.
Three accounting traps catch strong engineers.
First, the multiplier applies to the total the driver reports through torch.cuda.mem_get_info, not to the 80 GB on the box label, and the log reports GiB. 35.28 GB of weights shows up as 32.85 GiB, which looks like a mismatch and is not.
Second, CUDA graph memory is captured after the KV pool is allocated, because the graphs bind to the pool's addresses. Whether its size is counted inside the 0.92 has changed across vLLM releases, so on the version you run, compare the graph-capture log line against the remaining headroom. When it was not counted, the symptom was an instance that used more than its fraction, which breaks two engines sharing one GPU.
Third, the activation peak is measured once, at startup, for the configuration given. Raise max_num_batched_tokens from 8,192 to 32,768 and the MLP intermediates grow 4x to about 2.8 GB; load a LoRA adapter later or switch on a feature the profile did not exercise and the allocation comes from the 8% headroom. That headroom is why OOMs under load usually trace to a configuration change after profiling, not to the KV pool.
The reversal condition: with BF16 weights the same two GPUs hold 70.55 GB of weights each, and the budget leaves 3.05 GB for overheads and KV combined. At that point the accounting answer is "this does not fit", and the move is TP=4 or FP8, not a higher utilization.
What interviewers probe next
"Two vLLM instances share one GPU at 0.45 each. What breaks?" Each profiles assuming the other's memory is fixed, so start them sequentially, and any memory one instance allocates outside its budget (graphs, late allocations) comes out of the other's headroom.
"Where do NCCL buffers really go at TP=8?" Each communicator allocates per-channel buffers for every peer, and vLLM's custom all-reduce adds IPC-registered buffers. It is hundreds of MB, not GB, and it shows up as non-torch memory in the profile.
"Prefix caching is on. Does it change the accounting?" No. Cached blocks live inside the same pool and are evictable, so they consume capacity that would otherwise sit free and never cause an OOM.
How it's scored
Weak: subtracts weights from 80 GB and calls the rest KV, with no budget fraction, no activations and GB and GiB mixed.
Hire: applies the 0.92 budget, sizes weights per GPU under TP, reserves an activation peak tied to the token budget, and converts the remainder to tokens with per-GPU KV heads.
Strong Hire: describes the startup sequence (load, profile, non-torch accounting, allocate KV, capture graphs), says which figures come from the log, and explains why a post-profiling change is the usual source of an OOM.
Common mistakes
- Using the full 327,680 B per token at TP=2. Each GPU stores 4 of the 8 KV heads, so per-GPU tokens double.
- Treating activation memory as proportional to the number of running sequences. It tracks the tokens processed in one forward pass, which chunked prefill caps.
- Raising
gpu_memory_utilizationto fix preemptions on a box where a sidecar or a second process also allocates on the GPU. - Forgetting the logits buffer: 256 sequences times a 128,256-token vocabulary in FP32 is 131 MB before the sampler copies it.