Question 1

Derive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.

Accepted Answer

The formula takes one line. The answer interviewers score is the memory budget around it: what the engine reserves first, what tensor parallelism does to the heads, and why the count you get is a ceiling.

Question 2

Why does a paged KV cache waste at most one block per sequence, and how would you pick the block size?

Accepted Answer

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Question 3

Account for every gigabyte on a GPU running vLLM: weights, KV pool, activations, CUDA graphs and NCCL buffers.

Accepted Answer

Weights and KV are the two numbers everyone knows. The answer that gets the offer accounts for the other few gigabytes, says which ones vLLM measures and which it guesses, and knows where an OOM comes from when the sum looks fine.

Question 4

FP8 KV cache: what scales does it need, where is the error largest, and how would you validate it on long-context tasks?

Accepted Answer

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Question 5

Tensor parallelism beyond the KV head count replicates the cache. Derive the per-GPU KV at TP=16 for an 8-KV-head model.

Accepted Answer

Doubling the GPUs past the KV head count halves the weights per GPU and leaves the KV per GPU exactly where it was. The answer derives that, prices the duplicate copy, and picks the parallelism that does not pay it.

Question 6

A model mixes sliding-window and global attention layers. Compute its KV footprint at 128K and tell me what the window actually saves.

Accepted Answer

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Question 7

Design KV offload to CPU memory. When does fetching blocks back over PCIe beat recomputing them?

Accepted Answer

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

Question 8

Prefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?

Accepted Answer

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Question 9

Why can MLA decode attend over the 512-dimensional latent without decompressing keys, and what does absorption cost in FLOPs?

Accepted Answer

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

Question 10

With parallel sampling at n=8, a 2K-token prompt and 500 output tokens each, how many KV blocks does copy-on-write save?

Accepted Answer

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

Question 11

The KV pool is full and requests keep arriving. Compare admission control, preemption and spilling, and what each does to p99.

Accepted Answer

All three policies keep the server alive. They differ in who waits, who redoes work, and which percentile absorbs the damage, and the numbers for a 6K-token victim make the choice for you.

Question 12

In prefill-decode disaggregation, how does KV transfer size scale, and what link bandwidth keeps the transfer off the critical path?

Accepted Answer

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

Question 13

A model's config says max context 1M. Size the KV cache for a single 1M-token request on a GQA model and on an MLA model.

Accepted Answer

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Question 14

When you quantize the KV cache, why are keys more sensitive than values, and what per-channel or per-token scheme follows from that?

Accepted Answer

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

Question 15

Design a multi-tier KV cache across HBM, host DRAM and SSD for a fleet with heavy prefix reuse, and show the hit-rate math.

Accepted Answer

Each tier is worth exactly the recompute it avoids minus the fetch it costs. Work that out per token and the design writes itself, including the tier that should not exist for small models and the SSD that wears out in a week.

Question 16

A hybrid model mixes attention and recurrent layers. Budget its state and design safe prefix reuse and rollback.

Accepted Answer

A fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.

KV Cache & GPU Memory

KV Cache Interview Questions