Practice tests · 30 items

KV Cache & GPU Memory practice test

The KV algebra across MHA, GQA, MQA and MLA, block tables and fragmentation, hash and radix prefix reuse, eviction, FP8 KV, CPU and NVMe offload, and the gpu_memory_utilization budget that decides how many sequences actually fit. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test

Topic

Difficulty

Questions

10 in this pool · about 7 min

Reveal answers

Sample items with answers

easy · sample

Llama 3.3 70B has 80 layers, 64 query heads, 8 KV heads and head dimension 128. How many bytes of KV cache does one token occupy in BF16?

2,560 KiB, from 2 x 80 x 64 x 128 x 2 B

160 KiB, from 80 x 8 x 128 x 2 B

✓320 KiB, from 2 x 80 x 8 x 128 x 2 B

4 KiB, from 2 x 8 x 128 x 2 B

Per token the cache holds a K vector and a V vector for every KV head in every layer: 2 x 80 x 8 x 128 x 2 B = 327,680 B = 320 KiB. Using the 64 query heads is the classic GQA mistake and inflates the answer 8x; query heads share the 8 KV heads in groups of 8. Dropping the leading 2 counts only K or only V. 4 KiB is one layer's worth. At 8,192 tokens the 320 KiB becomes 2.5 GiB per sequence, the number every capacity plan starts from.

easy · sample

What does the KV cache store during autoregressive decode, and what does it save?

The final hidden state of every past token, so later steps rerun only the model's last layer

The attention probability matrix from the previous step, so softmax runs only once per sequence

✓Each past token's key and value vectors in every layer, so decode never recomputes them

The logits of past positions, so sampling can reuse them for repeated tokens

Each new token's query attends to the keys and values of all earlier tokens in every layer. Those K and V vectors depend only on earlier tokens, so caching them turns each decode step from reprocessing the whole sequence into processing one token plus a read of the cache. Attention probabilities change with every new query, so caching them would be useless. A final hidden state cannot rebuild the per-layer K and V without rerunning the layers that produced them. The cost of the cache is memory that grows linearly with context.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: