FlashAttention-2 reaches well under half of an H100's BF16 peak. The answer explains why in one ratio, exponentials against matmuls, and maps each FA3 technique to the stall it hides.
an account raises the free count per topic
← Attention Kernels / 09
expertNewNVIDIAOpenAIAnthropic
FlashAttention-3 on Hopper: explain warp specialization, TMA and ping-pong scheduling, and which bottleneck each one removes.
FlashAttention-2 reaches well under half of an H100's BF16 peak. The answer explains why in one ratio, exponentials against matmuls, and maps each FA3 technique to the stall it hides.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackFP8 attention: where does the precision break, and how do incoherent processing and per-block scales help?Next in this trackSliding-window attention in a tiled kernel: which blocks get skipped, and what speedup follows at 32K context with a 4K window?Next in this trackShould RoPE run inside the attention kernel or as a separate pass? Count the bytes saved, then find the precision trap at long positions.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.