hardNewNVIDIAOpenAIAnthropic

Could you keep the KV cache of hot sequences resident in the H100's 50 MB L2 during decode, and would it be worth it?

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackYou are about to report one number for your new kernel in a design review. How do you measure it so the number survives scrutiny?Next in this trackA kernel runs 20% slower on an H200 than on an H100, even though the H200 has 43% more memory bandwidth. How can that happen?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.