Skip to content
← KV Cache & GPU Memory / 19
hardNewAnthropicPerplexityBaseten

After enabling CPU KV offload, requests that hit the host tier have worse TTFT than requests that miss. Find out why.

The copy takes 2.7 ms and the link is healthy, yet a cached 1K-token prefix reaches its first token a full 100 ms after a cold one. The answer finds the cost the transfer metric cannot see and sets the prefix length below which a hit should be recomputed.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The copy takes 2.7 ms and the link is healthy, yet a cached 1K-token prefix reaches its first token a full 100 ms after a cold one. The answer finds the cost the transfer metric cannot see and sets the prefix length below which a hit should be recomputed.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackDeepSeek says V4.1-Flash keeps 890 bytes of KV per token. vLLM allocates 1,056. Reconcile the two numbers.Next in this track64 requests share a 2,000-token system prompt and hit a cold replica at once. How many times is the prefix computed?Next in this trackvLLM on a DGX Spark starts cleanly, then the desktop stalls and the trainer is OOM-killed. How do you budget memory?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.