gpu memory

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

141 GB of weights on 160 GB of GPUs, and the engine refuses to start. The answer walks the five phases of engine startup, shows which one failed from the log, and does the arithmetic that says the model never fit the way it was configured.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.