07Design KV offload to CPU memory. When does fetching blocks back over PCIe beat recomputing them?▼hardNewAnthropicPerplexityBaseten○ sign inRecomputing a prefix costs FLOPs proportional to the model, fetching it costs bytes proportional to the cache. One ratio decides it, and the answer shows by how much fetching wins and the three situations where it still loses.Open full answer →
04Preempt by recompute or by swapping KV to host memory? Derive the break-even from PCIe bandwidth and prefill speed.▼mediumNewAnyscaleRed HatDatabricksunlockedPer token, moving KV over PCIe is several times cheaper than recomputing it, yet vLLM recomputes. The answer derives the break-even bandwidth, shows how small copies fall below it, and names what makes recompute nearly free.Open full answer →