Skip to content
← KV Cache & GPU Memory / 21
hardNewAnthropicOpenAIFireworks AI

64 requests share a 2,000-token system prompt and hit a cold replica at once. How many times is the prefix computed?

Both engines have a prefix cache, both start empty, and both see the same 64 requests at once. One computes the shared prompt once, the other nearly seven times, and the difference is not the cache but the moment a new prefix becomes visible to the next request.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Both engines have a prefix cache, both start empty, and both see the same 64 requests at once. One computes the shared prompt once, the other nearly seven times, and the difference is not the cache but the moment a new prefix becomes visible to the next request.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackvLLM on a DGX Spark starts cleanly, then the desktop stalls and the trainer is OOM-killed. How do you budget memory?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDecode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.