01Derive online softmax, prove it is exact, and explain why it makes FlashAttention fast without reducing FLOPs.▼medium★ EssentialNewNVIDIAOpenAIAnthropicunlockedFlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.Open full answer →
04Count the HBM bytes FlashAttention reads and writes for one head at N = 8K and d = 128, and compare it with naive attention.▼mediumNewNVIDIAAnthropicMetaunlockedThe textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.Open full answer →