A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
A prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.
Nobody has commented on this answer. Add a correction or another way to work it.