TL;DR: Compare active page demand with the measured KV pool, including output growth and rounding. In this synthetic case, 200 prompts fit initially but cannot all reach their output limit: 25,600 pages grow to 38,400 against a 26,483-page pool; only 137 uniform full-length requests fit before safety margin.
How to approach it
Read the startup pool size, page size and actual cache layout. Correlate active page ownership, request lengths, preemption deltas and queue wait on the same replica. Check whether capacity shrank or demand grew. A full pool with useful reusable blocks is different from repeated eviction and recomputation on the critical path.
Inspect the evidence first. These are synthetic inputs for a dense GQA engine, not measurements or an engine default. There is no prefix sharing, speculative lookahead or swapping in this fixture. The hypothetical 32-layer model has 8 KV heads, head dimension 128 and BF16 K/V.
GPU budget: 80e9 B x configured fraction 0.92 = 73.6e9 B
weights: 16.06e9 B (assumed checkpoint allocation)
other reserved: 2.0e9 B (assumed complete non-KV allowance)
available KV: 73.6e9 - 16.06e9 - 2.0e9 = 55.54e9 B
KV bytes/token: 2 x 32 x 8 x 128 x 2 = 131,072 B
page size: 16 tokens x 131,072 B = 2,097,152 B
usable pages: floor(55.54e9 / 2,097,152) = 26,483
workload: 200 simultaneous 2,048-token prompts, up to 1,024 new tokens
Check yourself
Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.
1The fixture admits 200 requests of 2,048 tokens into 26,483 pages of 16 tokens. Can they all grow to 3,072 tokens? Compute both page demands and the fully grown concurrency ceiling.
2Changed constraint: the pool is full with only 12 running requests, and the configured cap is 200. What should you inspect next?
A strong answer
The sequence can pass admission and fail during growth. Free capacity at admission does not reserve the output's future pages. In this fixture, after every request reaches 2,112 cached tokens, each owns 132 pages. That consumes pages, leaving 83. Growing all 200 requests by one token would need 200 more pages, so some must wait, finish or be preempted.
Use the logged page count in a real server. A hand budget must include graph pools, activation peaks, workspaces and runtime allocations; a different startup profile can change usable KV even when weights and device capacity are unchanged. vLLM v0.31.0's preemption guidance, checked October 17, 2026, describes recomputation when KV capacity is insufficient and configuration changes that trade concurrency for memory.
This Python 3 page ledger deliberately uses no sharing. Run it to check the admission and full-growth cases:
def pages_needed(lengths, page_tokens=16):
if page_tokens < 1 or any(n < 0 or int(n) != n for n in lengths):
raise ValueError("nonnegative token counts and positive page size required")
return sum((n + page_tokens - 1) // page_tokens for n in lengths)
pool_pages = 55_540_000_000 // (16 * 131072)
initial = pages_needed([2048] * 200)
full = pages_needed([3072] * 200)
ceiling = pool_pages // pages_needed([3072])
print(f"pool {pool_pages}; initial {initial}; full {full}; ceiling {ceiling}")
print("partial-page check:", pages_needed([17, 17]), "pages")
Expected output is pool 26483; initial 25600; full 38400; ceiling 137. Two independent 17-token sequences need four pages, even though their combined 34 tokens fit in three pages if treated as one sequence. With sharing, count distinct physical pages and reference counts instead of summing logical tables.
| Evidence | Interpretation to test |
|---|---|
| Pool unchanged, active token demand rises | Admission/growth mismatch |
| Same workload, smaller startup pool | Changed graph/workspace/reserve accounting |
| Few requests own most pages | Long-context or output-tail concentration |
| Preemptions rise but recompute remains small | Reuse may recover victims cheaply |
| Pages stay owned after requests finish | Investigate lifecycle and allocator invariants |
A cap of 128 is a candidate here: pages, leaving pages. It is not a universal default or a guarantee for a different length mix. Measure the new queueing delay, rejection rate and SLO-qualified completions on the original offered load. Lower preemption with worse goodput is not a completed fix.
FP8 halves K/V payload bytes, but scale metadata, page allocation and the backend's workspace determine the realized capacity gain. Validate quality at the served context lengths before relying on it. Increasing the memory fraction is only defensible after checking the measured non-KV peak; consuming the last reserve can exchange recomputation for an OOM.
What interviewers probe next
"Why not reserve every request's maximum output?" It avoids this growth mismatch under the modeled accounting but can strand pages for requests that stop early; compare that policy against statistical admission with an explicit tail-risk budget.
"Every preemption costs a full prefill?" Only if the required KV was actually evicted; distinguish a preemption counter from measured recomputed tokens.
"A cap removes preemptions but the queue keeps growing." Offered work exceeds sustainable capacity at that setting; add capacity or enforce admission rather than hiding overload in a longer queue.
How it's scored
Weak: raises concurrency because the waiting queue is long, or treats 100% cache occupancy as proof of failure.
Hire: computes per-sequence page rounding, checks growth beyond admission and proposes a bounded concurrency change.
Strong Hire: reconciles physical ownership with the pool, distinguishes recomputation from recoverable reuse, and evaluates the fix by goodput and queue stability.
Common mistakes
- Rounding only after summing all sequence lengths.
- Planning from prompt lengths while omitting output growth.
- Assuming the configured request cap is always reached.
- Treating a page budget derived from rounded GB as exact token capacity.