Skip to content
← Performance Debugging & Benchmarking / 23
hardNewAnthropicOpenAIFireworks AI

Prefix-cache hit rate drops from 70% to 40% at peak. Is the cache too small, or is the router scattering prefixes?

The hit rate is fine at night and collapses at peak. The answer turns cache capacity into a survival time in seconds, measures how often each prefix returns to each replica, and shows why the histogram you would reach for hides the evidence at exactly the wrong moment.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The hit rate is fine at night and collapses at peak. The answer turns cache capacity into a survival time in seconds, measures how often each prefix returns to each replica, and shows why the histogram you would reach for hides the evidence at exactly the wrong moment.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackA rollout replay reports a 1.25 probability ratio before any weight update. What would you check first?Next in this trackA load test says TTFT got faster when we doubled concurrency. What is wrong with the benchmark?Next in this trackAn FP8 KV cache run is 3.4% faster than the BF16 run. Design the A/B that decides whether to ship it.
EDITOR'S NOTE

Strong candidates convert blocks into seconds: free cached blocks over the block allocation rate. Weak ones argue about cache size in gigabytes without a rate.

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.