Skip to content
← Attention Kernels / 08
mediumNewMetaMistral AINVIDIA

GQA in the attention kernel: how do you load each KV head once for its whole group of query heads?

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackFlashAttention-3 on Hopper: explain warp specialization, TMA and ping-pong scheduling, and which bottleneck each one removes.Next in this trackFP8 attention: where does the precision break, and how do incoherent processing and per-block scales help?Next in this trackSliding-window attention in a tiled kernel: which blocks get skipped, and what speedup follows at 32K context with a 4K window?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.