08Prefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?▼mediumNewAnthropicPerplexityFireworks AI○ sign inA prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.Open full answer →
04Walk through how vLLM's prefix caching identifies a reusable KV block and hands it to a new request, step by step.▼mediumNewAnyscaleRed HatDatabricksunlockedA block's identity is a hash of its tokens and of every block before it. The answer traces one request through hashing, lookup, the last-token rule and eviction, with a runnable model of the lookup.Open full answer →
05SGLang's radix cache versus vLLM's block hashing: compare the data structures, the eviction, and the workloads where each one wins.▼hardNewDatabricksxAI (SpaceXSI)Together AIunlockedBoth reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.Open full answer →