06A model mixes sliding-window and global attention layers. Compute its KV footprint at 128K and tell me what the window actually saves.▼hardNewGoogle DeepMindOpenAIMistral AI○ sign inHalf the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.Open full answer →
16A hybrid model mixes attention and recurrent layers. Budget its state and design safe prefix reuse and rollback.▼hardNewNVIDIATogether AIFireworks AI◆ premiumA fixed recurrent state still costs memory for every live request and saved checkpoint. Work through a state inventory, then find a restart boundary that every layer can actually restore.Open full answer →