l2 cache

The textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

A kernel time is a distribution measured under conditions, and a reviewer will ask about both. The answer builds the timing loop, says what each line defends against, and adds the one sanity check that catches most bad numbers.