Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.
an account raises the free count per topic
← KV Cache & GPU Memory / 06
hardNewGoogle DeepMindOpenAIMistral AI
A model mixes sliding-window and global attention layers. Compute its KV footprint at 128K and tell me what the window actually saves.
Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackDesign KV offload to CPU memory. When does fetching blocks back over PCIe beat recomputing them?Next in this trackPrefix-cache eviction: LRU by block or by request? And what does a radix tree change about it?Next in this trackWhy can MLA decode attend over the 512-dimensional latent without decompressing keys, and what does absorption cost in FLOPs?
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.