08GQA in the attention kernel: how do you load each KV head once for its whole group of query heads?▼mediumNewMetaMistral AINVIDIA○ sign inGQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.Open full answer →
05Why do tensor core GEMMs use tiles like 128 by 128 by 64, and what limits the tile size on Hopper?▼hardNewNVIDIAAMDOpenAIunlockedEvery dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.Open full answer →