08Prefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?▼mediumNewAnthropicPerplexityFireworks AI○ sign inA prefix cache only hits from the first token forward, so evicting the wrong block can strand gigabytes that stay resident and can never be hit. The answer derives how much, and which eviction order guarantees it never happens.Open full answer →
05SGLang's radix cache versus vLLM's block hashing: compare the data structures, the eviction, and the workloads where each one wins.▼hardNewDatabricksxAI (SpaceXSI)Together AIunlockedBoth reuse KV for shared prefixes, and on a 2K-token chat prompt their hit rates differ by under one percent. The answer finds the differences that do matter, from eviction policy to whether a cache key means anything outside one process.Open full answer →
06Implement a radix tree prefix cache: match, insert with edge splitting, request locks and LRU eviction of unlocked leaves.▼hardNewAnthropicOpenAIPerplexity○ sign inMatching a prefix is a trie walk. The grade is in the edge split, which cached tokens a running request pins, and an eviction loop that turns parents into leaves without ever freeing a slot twice.Open full answer →