vLLM fails to start with an out-of-memory error on a model that should fit. Walk through the startup memory profile.

TL;DR: "Fits" has to mean weights plus activation peak plus runtime overhead plus KV for at least one max_model_len sequence, all inside gpu_memory_utilization of each GPU. Llama 3.3 70B in BF16 on two H100s puts 70.55 GB of weights on each GPU against a 73.6 GB budget at the 0.92 default (vLLM v0.31.0; 0.90 through v0.19), which leaves 3.05 GB for everything else, and one 131,072-token sequence needs 21.5 GB of KV per GPU. Find the phase that failed in the startup log, then fix the term that is too big: FP8 weights free 35 GB per GPU.

How to approach it

Ask for the exact error and the last log lines before it, because the phase that failed decides the fix. Then ask for the model, dtype, tensor-parallel size, max_model_len, max_num_batched_tokens and gpu_memory_utilization, and whether anything else holds memory on those GPUs. Do the per-GPU arithmetic before touching a flag.

A strong answer

Engine startup is a sequence, and each phase has its own failure:

The KV pool in phase 4 is whatever remains of the budget after phases 1 to 3:

$$\text{KV}_{\text{gpu}} = u \cdot M_{\text{gpu}} - W_{\text{gpu}} - A_{\text{peak}} - R_{\text{non-torch}}$$

Apply it to the configuration that "should fit":

Llama 3.3 70B, BF16, TP=2 on H100 SXM 80 GB, u = 0.92
weights per GPU = 70,553,706,496 x 2 B / 2      = 70.55 GB
budget per GPU  = 0.92 x 80 GB                   = 73.60 GB
left for activations, runtime and KV             =  3.05 GB
KV per token per GPU = 327,680 B / 2             = 163,840 B
one 131,072-token sequence = 131,072 x 163,840 B = 21.47 GB per GPU
sanity: 141 GB total < 160 GB, and it still cannot hold a single full sequence

The intuition "141 is less than 160" ignores three terms. The budget is a fraction of the GPU, not all of it. The profile run in phase 3 executes a dummy forward at max_num_batched_tokens and records the activation peak: the MLP's gate and up outputs alone at 8,192 tokens are 8,192 x 14,336 x 2 x 2 B = 470 MB per GPU. And the non-torch memory (CUDA context, NCCL buffers, the allocator's fragmentation) is real and varies by driver and version; read it from the log rather than assuming it. With 3.05 GB left before those terms, the pool comes out at a gigabyte or less, a twentieth of what one full-length sequence needs, and the engine stops with an error that it has no memory for KV cache blocks, or that max_model_len exceeds what the cache can store. People report both as "OOM", but they are capacity checks, not allocator failures.

Map the symptom to a hypothesis, ranked by how often each explains "should fit":

The single check that separates the top two is nvidia-smi --query-gpu=memory.used --format=csv on each GPU before launch, together with whether the "model loading" log line appeared. Memory already used by a stale process or a second engine makes phase 2 fail; a clean GPU with weights loaded and a KV error points at phase 4. vLLM V1 workers also compare free memory against the requested fraction at startup and refuse to start if it is short; check the exact message in your version.

Then fix the term that is wrong, with numbers:

option A: u = 0.95, BF16  -> 76.0 - 70.55 = 5.45 GB before activations and runtime
          with an illustrative 1.5 GB for those: 3.95 GB / 163,840 B = ~24,000 tokens
          fits max_model_len 16,384 (2.68 GB), not 131,072
option B: FP8 weights, u = 0.92 -> 73.6 - 35.28 - 1.5 = 36.8 GB -> ~224,700 tokens
option C: TP=4, BF16 -> 35.28 GB per GPU, same ~37 GB pool per GPU, twice the GPUs

Option B is usually the right call for serving: it fits a full-length sequence with room for concurrency and also halves the per-step weight reads. Option C is the answer when FP8 fails the quality bar, at twice the GPUs per replica. Option A only works if the product can live with 16K contexts.

The reversal condition: if the KV pool is large and the engine dies only in phase 5, the arithmetic is fine and the headroom is not. Graph capture allocates after the pool is sized, and whether its memory is counted inside the budget has changed across releases, so lower u by a few points or reduce the captured batch sizes rather than shrinking the model.

What interviewers probe next

"Why not set gpu_memory_utilization to 0.99?" Graph capture, NCCL and fragmentation need room outside the pool; at 0.99 the engine starts fine on a quiet node and dies later, or fails at capture.

"It starts, but with only 3,000 tokens of KV. Is that a problem?" Yes: concurrency equals pool tokens over mean context, so at 3,000 tokens the engine runs one or two requests and preempts constantly.

"Two engines share one GPU with u = 0.45 each. What goes wrong?" The fraction applies to total memory, so each engine's runtime overhead and graphs come out of a budget that was sized assuming it was alone.

How it's scored

Weak: compares total weights to total GPU memory and suggests "add a GPU" without locating the failing phase.

Hire: knows the profile-then-allocate sequence, does per-GPU arithmetic with the utilization fraction, and checks that one max_model_len sequence fits.

Strong Hire: separates allocator OOMs from KV capacity errors by phase, names the pre-launch check that splits the top two hypotheses, and prices each fix in tokens of KV.

Common mistakes

• Forgetting that gpu_memory_utilization is a fraction of total memory, not of free memory.
• Sizing KV with the total per-token figure on a TP=2 replica instead of half of it per GPU.
• Raising the utilization fraction to fix a phase 5 failure, which makes it worse.
• Mixing GB and GiB: the engine log reports GiB, and 73.6 GB is 68.5 GiB.