KV cache chains: bytes per token, the pool, and what reuse saves

Every KV question in a loop starts from one of these. Each chain names its model and part, shows the substitution, and ends with a line that tells you whether the result is plausible. The linked questions carry each chain into a decision.

Bytes per token, GQA

inputs   Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128, BF16 (2 B)
         Llama 3.3 70B: 80 layers, 8 KV heads, head dim 128, BF16
formula  2 (K and V) x layers x KV heads x head dim x bytes
8B       2 x 32 x 8 x 128 x 2 B = 131,072 B = 128 KiB per token
70B      2 x 80 x 8 x 128 x 2 B = 327,680 B = 320 KiB per token
sanity   one 8,192-token 8B sequence = 8,192 x 128 KiB = 1 GiB

The usual slip is multiplying by query heads. Llama 3.1 8B has 32 of them, so that answer is 4x too large and every capacity figure after it is wrong by the same factor.

Drilled in: KV footprint for MHA, GQA and MLA · KV cache bytes and budget

Bytes per token, MLA

inputs   DeepSeek V3: 61 layers, BF16; per layer one 512-wide latent
         and one 64-wide RoPE key
formula  layers x (latent + rope key) x bytes
         61 x (512 + 64) x 2 B = 70,272 B per token
sanity   70,272 / 131,072 = 0.54: a 671B model caches about half
         of what Llama 3.1 8B does per token

The latent replaces both K and V, so there is no factor of 2. Forgetting the 64-wide RoPE key understates the cache by 11%. Under tensor parallelism the latent cannot be split by head, so each rank holds all of it.

Drilled in: DP attention plus EP for MLA models

The pool left for KV

inputs   H100 SXM 80GB: CUDA reports 79.18 GiB = 85.02 GB
         vLLM v0.32.0 gpu_memory_utilization 0.92 (default)
         Llama 3.1 8B BF16 weights 16.06 GB
         activations and other allocations assumed 3.0 GB
budget   0.92 x 85.02 GB = 78.22 GB
pool     (78.22 - 16.06 - 3.0) GB / 131,072 B = 451,355 tokens
sanity   451,355 / 8,192 = 55 sequences of 8K tokens at once

Two errors cost the most here: starting from the 80 on the label instead of what CUDA reports, and mixing GB with GiB, which is a 7% error. The 3.0 GB is an assumption; the engine's startup log reports the real figure, and that is the number to use.

Drilled in: vLLM memory accounting

What block rounding wastes

inputs   vLLM block of 16 tokens (v0.32.0 default on CUDA); 200 running sequences
         the 451,355-token pool above; lengths uniform modulo 16
per seq  at most 15 tokens in the last block, 7.5 on average
total    200 x 7.5 = 1,500 tokens = 0.33% of the pool (197 MB at 128 KiB)
256-token blocks  200 x 127.5 = 25,500 tokens = 5.6% of the pool
sanity   paging made waste small; block size is the knob that brings it back

Paging removes the contiguous reservation, not the partial block. Larger blocks help the kernel and hurt capacity in direct proportion.

Drilled in: Paged KV waste bound

Sliding-window layers

inputs   gpt-oss-120b: 18 global layers and 18 window-128 layers
         8 KV heads x 64 dims x 2 (K, V) x 2 B = 2,048 B per layer per token, BF16
128K     global 18 x 131,072 x 2,048 B = 4.83 GB; window 18 x 128 x 2,048 B = 4.7 MB
total    4.84 GB, against 36 x 131,072 x 2,048 B = 9.66 GB with every layer global
sanity   2.0x, the 1:1 layer ratio; the 128-token window barely appears

The saving is set by how many layers are windowed. Whether the engine actually frees the windowed layers' old blocks is a separate question about its allocator.

Drilled in: Sliding-window hybrid KV at 128K

A shared system prompt

inputs   2,000-token system prompt, 100 concurrent requests, Llama 3.1 8B BF16
         H100 SXM 989 TFLOP/s dense BF16, 50% MFU assumed
memory   unshared 100 x 2,000 x 131,072 B = 26.2 GB; shared once = 262 MB
compute  each hit skips 2 x 8.03e9 x 2,000 = 32 TFLOP = 65 ms of prefill
blocks   2,000 / 16 = 125 full blocks, so the whole prompt is shareable
sanity   a hit saves pool and TTFT together; a 2,007-token prompt
         shares 125 blocks and recomputes 7 tokens

vLLM matches whole blocks only, and a block's key chains its parent's hash, so one changed token near the start of the prompt invalidates every block after it.

Drilled in: Prefix caching and block hashing

Fetch from host memory or recompute

inputs   8,192-token prefix, Llama 3.1 8B BF16; PCIe Gen5 at an assumed 50 GB/s
         recompute 1.49e14 FLOPs (linear plus attention) at 60% of 989 TFLOP/s
bytes    8,192 x 131,072 B = 1.07 GB
fetch    1.07 GB / 50 GB/s = 21.5 ms
compute  1.49e14 / (0.6 x 989e12) = 251 ms
sanity   fetching wins when bytes per token / link bandwidth
         < FLOPs per token / achieved compute

The copy is the part a transfer metric reports. A hit's TTFT also includes the lookup, the scheduling of the load and whatever the engine does before the first step, so time the hit path end to end before deciding where the break-even sits.

Drilled in: KV offload: fetch or recompute · Host-tier hits slower than misses

FP8 KV

inputs   the pool above, KV stored as FP8 E4M3 (1 B) instead of BF16
per token  2 x 32 x 8 x 128 x 1 B = 64 KiB
pool       59.16 GB / 65,536 B = 902,710 tokens, twice the BF16 pool
sanity     the weights did not shrink, so the KV budget is unchanged
           and the token count doubles exactly

The capacity arithmetic is the easy half. vLLM uses a scale of 1.0 when a checkpoint carries no KV scales and warns once at startup, so read the log before trusting the accuracy of a doubled pool.

Drilled in: FP8 KV scales and long-context validation