01Derive online softmax, prove it is exact, and explain why it makes FlashAttention fast without reducing FLOPs.▼medium★ EssentialNewNVIDIAOpenAIAnthropicunlockedFlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.Open full answer →
03Implement online softmax for a whole tile of queries, not one query, and write the rescaling step in matrix form.▼mediumNewNVIDIAOpenAIAnthropicunlockedThe one-query recurrence is a warm-up. A real kernel carries a vector of maxima and a vector of sums for a tile of rows, and the rescale becomes a diagonal matrix. The answer writes it, runs it, and shows the edge case that turns it into NaN.Open full answer →
11Sliding-window attention in a tiled kernel: which blocks get skipped, and what speedup follows at 32K context with a 4K window?▼mediumNewMistral AIOpenAIGoogle DeepMind◆ premiumA 4K window over 32K tokens keeps one eighth of the keys, and the speedup over causal attention is not 8x. The answer counts tiles exactly, classifies which need masking, and explains where the rest of the gap goes.Open full answer →
05Why do tensor core GEMMs use tiles like 128 by 128 by 64, and what limits the tile size on Hopper?▼hardNewNVIDIAAMDOpenAIunlockedEvery dimension of a GEMM tile is pinned by a different piece of hardware. The answer derives the reuse that sets the M and N sides, the swizzle row that sets K, and the register and shared-memory budgets that stop the tile growing.Open full answer →