Derive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.

TL;DR: KV bytes per token are $2 \cdot L \cdot n_{kv} \cdot d_h \cdot b$, which is 320 KiB for Llama 3.3 70B in BF16 thanks to 8 KV heads instead of 64. On four H100s with BF16 weights and vLLM's 0.92 memory fraction, about 141 GB is left for cache, 431K tokens, so roughly 52 conversations held at the full 8K, or 105 with an FP8 cache.

How to approach it

Separate the two halves out loud: the per-token footprint is a property of the model's attention design, and the number of sequences is a property of the memory budget left after the engine has taken its share. Ask the precision of the weights and of the cache, the tensor-parallel degree, and whether "8K" means every conversation sits at 8K or that 8K is the cap. The last one changes the answer by 2x or more.

A strong answer

Each layer caches one key and one value vector per KV head per token, so:

$$\text{KV/token} = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot b$$

The three attention designs differ only in what $n_{kv} \cdot d_h$ is. Llama 3.3 70B has 80 layers, 64 query heads, 8 KV heads and head dimension 128; DeepSeek V3 is the MLA reference:

MHA (64 KV heads)  = 2 x 80 x 64 x 128 x 2 B = 2,621,440 B = 2.5 MiB
GQA (8 KV heads)   = 2 x 80 x  8 x 128 x 2 B =   327,680 B = 320 KiB    8x smaller
MQA (1 KV head)    = 2 x 80 x  1 x 128 x 2 B =    40,960 B =  40 KiB    64x smaller
MLA, DeepSeek V3   = 61 layers x (512 latent + 64 RoPE key) x 2 B = 70,272 B = 68.6 KiB

MLA caches a 512-dimensional latent per layer that all 128 heads decompress from, plus a small decoupled RoPE key. The RoPE part is kept separate because a position-dependent rotation sitting between the latent and the query would stop the key up-projection from being absorbed into the query projection, and that absorption is what lets decode attend over the latent directly. Caching DeepSeek V3's full per-head keys (192 wide) and values (128 wide) instead would cost 5.0 MB per token, 71x more. That ratio is why MLA models serve long contexts that a GQA model of the same size cannot.

Now the budget. vLLM sizes the cache last: it loads weights, runs a profiling forward pass at the largest batch it will schedule to measure peak activation memory, accounts for memory allocated outside PyTorch (NCCL buffers among it), and gives the cache whatever remains of gpu_memory_utilization times device memory. The default is 0.92 in vLLM 0.20 and later (checked against the v0.31.0 source, October 2026); it was 0.9 before that, and plenty of runbooks still say 0.9.

per-GPU budget  = 80 GB x 0.92                    = 73.6 GB
weights (TP=4)  = 141.1 GB / 4                    = 35.3 GB
reserved        = activations, graphs, NCCL (assume) = 3.0 GB
KV per GPU      = 73.6 - 35.3 - 3.0               = 35.3 GB
KV pool         = 4 x 35.3 GB                     = 141.3 GB
tokens          = 141.3e9 / 327,680               = 431,190 tokens
at 8,192 each   = 52.6  ->  52 conversations at full length
FP8 KV cache    = 862,381 tokens -> 105 conversations
MHA instead     = 53,898 tokens  ->  6 conversations

Two details make this exact rather than approximate. First, tensor parallelism splits KV heads, not tokens: at TP=4 each GPU holds 2 of the 8 KV heads for every token, which is why the per-GPU pools simply add. At TP=16, with only 8 KV heads, engines replicate heads and the pool stops growing with GPU count. Second, paging wastes at most one partially filled block per sequence: with 16-token blocks that is 15 tokens, 4.7 MiB, under 0.2% at 8K.

The count is a ceiling on simultaneous full-length sequences, not a concurrency target. Real conversations average far shorter than their cap, so the scheduler admits more and preempts when the pool runs dry. The capacity number tells you where preemption begins. It does not tell you the concurrency you will measure.

Now size 100 active requests at 32K. Define 32K as 32,768 resident tokens, including generated history. Use the same 80-layer, 8-KV-head, 128-dimensional GQA geometry above; “70B” alone does not specify those dimensions. This is an analytical payload count with no shared prefixes:

BF16 per request = 32,768 x 327,680 B = 10,737,418,240 B = 10 GiB
100 active       = 100 x 10 GiB = 1,000 GiB = 1,073.742 GB
FP8 payload      = 1,000 / 2 = 500 GiB, before scale metadata
MHA countercase  = 1,000 x (64 / 8) = 8,000 GiB at BF16

That BF16 KV payload alone exceeds four 80 GB GPUs, before weights or runtime memory. TP sharding can divide supported KV heads across devices; it cannot erase the aggregate payload. One hundred daily users is a different input from one hundred simultaneous full-context requests.

Check yourself

Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.

1. 1All 100 BF16 requests sit exactly at 32,768 tokens with 16-token pages. Each appends one token. How much physical KV payload must be added?

   Answer it, then reveal

2. 2Weights switch to FP8 but the KV cache stays BF16. Does the 1,000 GiB payload halve?

   Answer it, then reveal

The reversal condition: if contexts are short, say 1K tokens on average, the same pool holds 400-plus sequences and the limit moves to max_num_seqs, compute, or latency targets. KV capacity is the binding constraint for long-context and high-concurrency workloads, not for every workload.

What interviewers probe next

"Why not just set gpu_memory_utilization to 0.99?" The headroom absorbs allocations the profiling pass did not see: fragmentation, a larger prefill chunk, a LoRA adapter loaded later. Too little and the failure is a CUDA OOM mid-traffic rather than a preemption, which is much worse.

"The same model in FP8 weights: how much does the pool grow?" Weights drop to 17.6 GB per GPU, freeing 17.6 GB per GPU, so the pool grows from 141 GB to 212 GB, 50% more conversations, without touching the cache precision.

"Is an FP8 KV cache free?" Capacity doubles, but attention now reads quantized keys, and long-context retrieval tasks are where the error shows first. Evaluate on long-context tasks, not perplexity, and prefer per-head or per-token scales over one per-tensor scale.

"What does a hybrid model with sliding-window layers change?" Window layers cap at the window size per sequence, so only the global layers grow with context, and the per-token average falls as context grows.

How it's scored

Weak: multiplies by the 64 query heads instead of the 8 KV heads, or divides the GPU's total memory by the per-token size without subtracting weights.

Hire: derives 320 KiB, explains the GQA and MLA reductions, and subtracts weights and a reserve from a utilization-scaled budget to reach a token count.

Strong Hire: explains what the engine's profiling pass reserves and why, notes that TP splits KV heads and stops helping past the KV-head count, bounds paging waste, and separates the full-length ceiling from achievable concurrency.

Common mistakes

• Quoting vLLM's default as 0.9. It has been 0.92 since 0.20, and an interviewer who works on the engine will notice.
• Forgetting MLA's RoPE key: 512 instead of 576 per layer understates the cache by 11%.
• Reporting "52 concurrent users" as a capacity plan. It is the number at which every sequence at its maximum length exhausts memory, which real traffic rarely reaches all at once.
• Mixing GiB for the per-token size with GB for the budget, which drifts the token count by 7%.