Skip to content
← KV Cache & GPU Memory / 22
mediumNewNVIDIAQualcommMeta

vLLM on a DGX Spark starts cleanly, then the desktop stalls and the trainer is OOM-killed. How do you budget memory?

On a GPU with its own HBM, the memory fraction is a share of memory nobody else wants. On a unified-memory machine it is a share of the whole computer. The answer works out what vLLM's default leaves for the OS, how the engine measures free memory there, and how to size the pool from the work instead.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

On a GPU with its own HBM, the memory fraction is a share of memory nobody else wants. On a unified-memory machine it is a share of the whole computer. The answer works out what vLLM's default leaves for the OS, how the engine measures free memory there, and how to size the pool from the work instead.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDecode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.Popular right nowAccount for every gigabyte on a GPU running vLLM: weights, KV pool, activations, CUDA graphs and NCCL buffers.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.