12Implement beam search on a paged KV cache. How do forking and pruning work on block tables, and what does beam width 4 cost in memory?▼mediumNewGoogle DeepMindMetaNVIDIA◆ premiumFour beams over a 1,000-token prompt cost 600 MiB if each keeps its own cache and as little as 156 MiB if they share blocks. The answer implements the fork, the copy-on-write and the prune, then shows what vLLM actually does today.Open full answer →
11Implement beam search over a paged KV cache, forking beams by sharing blocks and copying only the tail on write.▼hardNewNVIDIAMetaGoogle DeepMind◆ premiumBeam search reshuffles which hypotheses survive every step, and a naive cache copies every survivor's history. The answer forks block tables instead, gets the release order right so the last sharer writes in place, and proves it against a beam search that never pages.Open full answer →