The KV Cache: Bytes per Token and the Memory Budget

TL;DR: Each token costs $2 \cdot L \cdot n_{kv} \cdot d_h \cdot b$ bytes of cache, 128 KiB for Llama 3.1 8B in BF16. On one 80 GB H100 at vLLM's 0.92 memory fraction, about 54.5 GB remains after weights and a reserve, which is 416K tokens: roughly 50 sequences held at a full 8K context.

What is stored and why

Attention at position $t$ needs the keys and values of every earlier position, in every layer. Those vectors depend only on the earlier tokens, so recomputing them each step would turn a decode step into a prefill over the whole history. The engine computes them once, during prefill or the step that produced the token, and keeps them in HBM. That store is the KV cache.

Per token and per layer it holds one key and one value for each KV head. Summed over layers:

The query head count does not appear. Grouped-query attention (GQA) lets a group of query heads read one KV head, which divides the cache by the group size. Multi-head latent attention (MLA, DeepSeek V2 and V3) goes further and caches one compressed latent per layer, shared by every head, plus a small decoupled RoPE key.

DeepSeek V3 has 671B parameters and the smallest cache in the table. That is the design point of MLA.

The budget

The cache gets what is left. vLLM loads the weights, runs a profiling forward pass at the largest batch it will schedule to measure peak activation memory, counts non-PyTorch allocations such as NCCL buffers, and hands the remainder of gpu_memory_utilization times device memory to the KV pool. That fraction defaults to 0.92 in vLLM v0.31.0 (read in vllm/config/cache.py at the tag, October 2026). It is a share of the whole device, so another process on the same GPU eats into it.

Worked example: Llama 3.1 8B on one H100

KV/token     = 2 x 32 layers x 8 KV heads x 128 x 2 B   = 131,072 B = 128 KiB
budget       = 80 GB x 0.92                             = 73.6 GB
weights      = 8.03e9 params x 2 B                      = 16.06 GB
reserve      = activations, CUDA graphs, NCCL (assumed) = 3.0 GB
KV pool      = 73.6 - 16.06 - 3.0                       = 54.54 GB
tokens       = 54.54e9 / 131,072                        = 416,103 tokens
at 8,192     = 50.8  ->  50 sequences at full length
FP8 cache    = 65,536 B/token -> 832,206 tokens -> 101 sequences
sanity       = one sequence at the 131,072-token max context is
               131,072 x 128 KiB = 16 GiB = 17.2 GB, more than the weights

The same arithmetic for Llama 3.3 70B with FP8 weights on one 141 GB H200 leaves 56.2 GB of pool, 171K tokens at 320 KiB each, about 20 sequences at 8K. The bigger model on the bigger GPU holds fewer conversations, because both its weights and its per-token cost grew.

Where it breaks

The count is a ceiling on simultaneous full-length sequences. Real requests stop short of their cap, so the scheduler admits more than 50 and preempts when the pool runs dry; capacity tells you where preemption begins, not what concurrency you will measure.

Tensor parallelism splits KV heads across GPUs, so pools add up linearly only while the TP degree is at most the KV-head count. Llama 3.1 8B has 8 KV heads; at TP=16 heads are replicated and the extra GPUs add weight capacity but not cache capacity.

The reserve is a measurement, and a wrong one fails late. If a later allocation the profiling pass did not see (a LoRA adapter loaded later, a larger prefill chunk, allocator fragmentation) needs memory, the result is a CUDA out-of-memory error mid-traffic rather than a clean preemption. That is why raising the fraction toward 0.99 is a poor trade for a few percent more cache.

Hybrid models change the per-token cost with length. Sliding-window layers cap at the window, and linear-attention layers keep a fixed-size state, so only the full-attention layers grow with context. Quoting one bytes-per-token number for such a model is wrong at every length but one. And an FP8 cache doubles capacity at an accuracy cost that shows up first on long-context retrieval, so it needs a long-context evaluation before it ships.

What interviewers listen for

The formula with KV heads, not query heads, and the units stated: the per-token size in KiB, the budget in GB, and the conversion done once. Then the budget built in the engine's order: utilization fraction, weights, measured reserve, pool. Strong answers name the GQA and MLA reductions as ratios, note where tensor parallelism stops adding cache, and separate the full-length ceiling from achievable concurrency.

Say it in 30 seconds

"The cache stores keys and values per layer per KV head per token: two times layers times KV heads times head dim times bytes, 128 KiB for Llama 8B in BF16. The engine gives the cache whatever is left of 92% of the GPU after weights and the activation reserve, about 54 GB on an H100, which is 416K tokens, or 50 sequences at 8K. An FP8 cache doubles that. That number is where preemption starts, not a concurrency promise."