prefix caching

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.

Each tier is worth exactly the recompute it avoids minus the fetch it costs. Work that out per token and the design writes itself, including the tier that should not exist for small models and the SSD that wears out in a week.

A fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.

With no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.

A block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.

Both reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

Matching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.

Two prompts with the same 16 tokens in block 3 must not share block 3 unless blocks 0 to 2 match too. The answer chains the hashes, keeps freed blocks hittable until memory is needed, evicts tails before heads, and proves a colliding hash cannot hand a request the wrong KV.