hardNewNVIDIATogether AIMeta

On a B200, why does FlashAttention-4 compute some exponentials in software and skip most softmax rescales? Show the arithmetic.

Blackwell doubled tensor-core work per SM per clock and left the exponential unit where it was. The answer counts cycles per tile, then shows why two tricks that look like approximations are safe.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Blackwell doubled tensor-core work per SM per clock and left the exponential unit where it was. The answer counts cycles per tile, then shows why two tricks that look like approximations are safe.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackgpt-oss adds a learned sink logit to every attention head. How does that change a fused attention kernel and its split-KV merge?Next in this trackWhat does DSA's lightning indexer plus top-2,048 selection cost at 128K context, and when does it beat dense attention?Next in this trackSparse MLA decode on a B200 reads FP8 or FP4 KV but multiplies in BF16. Is it bandwidth-bound, and why dequantize?

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.