TL;DR: Inspect tokenized prefixes and per-replica cache evidence before blaming kernels. The synthetic case below increases computed prompt work from 568 to 3,000 tokens/request, a 5.282x ratio; that can amplify queueing, but it does not by itself predict a p99 latency ratio.
How to approach it
Freeze the rollout timeline, workload mix and engine configuration. Decompose TTFT into queue wait and service time, then compare token IDs rather than visible prompt text. Check cache identity, replica placement and process restarts alongside the template diff. Flat GPU utilization does not prove unchanged useful work.
Inspect the evidence first. This is a synthetic incident fixture, not an observed deployment. Counters below are already differenced within one uninterrupted process epoch and one replica; names are normalized for this exercise.
| Same-duration window | Requests | Prompt tokens | Cached prompt tokens | Prompt length |
|---|---|---|---|---|
| Before template change | 100 | 300,000 | 243,200 | 3,000 each |
| After template change | 100 | 300,000 | 0 | 3,000 each |
The intended shared prefix is 2,560 tokens, followed by 440 request-specific tokens. The new serializer places a different request ID at the beginning. Router, model revision, cache salt, cache capacity and token count are unchanged in this fixture.
Check yourself
Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.
1In the two 100-request windows below, how many prompt tokens were recomputed per request before and after the change? What must be checked before using the ratio?
2Changed constraint: token IDs and cache identity are unchanged, but only the newly added replicas have zero hits. Which investigation comes first?
A strong answer
The cache key includes the preceding block's hash, current tokens and relevant extra identity. A differing token early in the sequence changes that block and its descendants, even if later text matches. vLLM v0.31.0 prefix-cache design, checked October 17, 2026, documents this chained identity and full-block reuse.
before: cached/request = 243,200 / 100 = 2,432 = 95% x 2,560
computed/request = 3,000 - 2,432 = 568
after: computed/request = 3,000 - 0 = 3,000
ratio: 3,000 / 568 = 5.282x
sanity: total prompt length did not change; reusable work did
A runnable Python 3 regression fixture checks the mechanism on already-tokenized synthetic prompts. Same model, adapter, salt and input identities are assumed. It checks reusable full-prefix blocks, not hash collision behavior or live cache residency:
def reusable_prefix(a, b, block=16):
if block < 1:
raise ValueError("positive block size required")
equal = 0
for left, right in zip(a, b):
if left != right:
break
equal += 1
return equal // block * block
shared = list(range(2560))
a = shared + [10001] + [7] * 439
b = shared + [10002] + [7] * 439
broken_a = [10001] + shared + [7] * 439
broken_b = [10002] + shared + [7] * 439
print("eligible prefix before:", reusable_prefix(a, b))
print("eligible prefix after:", reusable_prefix(broken_a, broken_b))
Expected output: 2560 before and 0 after. Eligibility does not imply a hit: the correct blocks must still be resident on the selected replica. For a real regression test, render the actual chat template, tokenize with the pinned tokenizer and compare those token IDs. Also compare adapter, multimodal input and cache-isolation keys; identical text alone is insufficient.
Distinguish the alternatives before the fix:
| Observation | Next falsifiable check |
|---|---|
| All replicas lose reuse after the same token diff | Replay old/new serialized prompts on one warm replica |
| Only new replicas miss, tokens unchanged | Check cache warmup and prefix-affinity routing |
| Hit rate falls after cache pressure rises | Inspect active blocks, eviction and preemptions |
| Reuse is unchanged but queue wait grows | Check arrival burst, admission limits and replica imbalance |
| Computed tokens unchanged but service slows | Profile backend, graph mode and kernel selection |
Compare counter deltas with matching labels and reset-aware windows; do not divide unrelated cumulative counters. A token hit fraction differs from the fraction of requests that hit anything. Avoid adding p99 queue time to p99 prefill time: use per-request traces because percentile sums are not generally valid.
Restore stable shared content before variable request fields where the prompt semantics permit it. Keep tenant isolation intact; never remove salts merely to recover hit rate. Canary the corrected template against identical traffic, confirm reused tokens recover, and check TTFT, token gaps and goodput. If reuse recovers but latency does not, continue into admission and service traces instead of declaring victory.
What interviewers probe next
"The first changed token is at offset 2,559 with 16-token pages." Only leading tokens are eligible as complete blocks.
"The same text hashes differently after an adapter change." Cache identity must include the adapter; those misses can be required for correctness.
"Will the 5.282x work ratio produce a 5.282x p99?" No. Prefix attention, batching and queueing alter the relationship; measure paired traces or replay the workload.
How it's scored
Weak: treats flat utilization as spare capacity or declares a kernel regression without inspecting the deploy diff.
Hire: calculates computed tokens/request, checks tokenized prefixes and separates cache loss from queueing.
Strong Hire: distinguishes eligible from resident blocks, rejects reset-contaminated metrics, preserves cache isolation and writes a boundary-sensitive regression test.
Common mistakes
- Comparing visible strings while ignoring token IDs or adapter/cache identity.
- Summing separate latency percentiles.
- Treating a warm single-replica replay as proof that fleet routing is healthy.
- Removing isolation keys to improve cache statistics.