Skip to content
← KV Cache & GPU Memory / 10
mediumNewOpenAIAnthropicFireworks AI

With parallel sampling at n=8, a 2K-token prompt and 500 output tokens each, how many KV blocks does copy-on-write save?

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Count the blocks with and without sharing and the saving looks large. The answer that scores also knows when the copy fires, what a misaligned prompt does, and the workload where the saving all but disappears.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.