Skip to content
← KV Cache & GPU Memory / 07
hardNewAnthropicPerplexityBaseten

Design KV offload to CPU memory. When does fetching blocks back over PCIe beat recomputing them?

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Recomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackPrefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?Next in this trackWhy can MLA decode attend over the 512-dimensional latent without decompressing keys, and what does absorption cost in FLOPs?Next in this trackWith parallel sampling at n=8, a 2K-token prompt and 500 output tokens each, how many KV blocks does copy-on-write save?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.