06A model mixes sliding-window and global attention layers. Compute its KV footprint at 128K and tell me what the window actually saves.▼hardNewGoogle DeepMindOpenAIMistral AI○ sign inHalf the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.Open full answer →
11Sliding-window attention in a tiled kernel: which blocks get skipped, and what speedup follows at 32K context with a 4K window?▼mediumNewMistral AIOpenAIGoogle DeepMind◆ premiumA 4K window over 32K tokens keeps one eighth of the keys, and the speedup over causal attention is not 8x. The answer counts tiles exactly, classifies which need masking, and explains where the rest of the gap goes.Open full answer →