05Why does FlashAttention-2's backward pass recompute attention, and what exactly does the forward pass save for it?▼hardNewOpenAIAnthropicGoogle DeepMindunlockedStoring the attention matrix for backward costs gigabytes per layer; recomputing it costs extra FLOPs. The answer derives the gradient, shows the one scalar per row that makes recomputation exact, and the identity that removes the last N-squared term.Open full answer →
06Split-KV decoding: how many splits would you use for batch 1 at 128K context on 132 SMs, and how are the partial results merged?▼hardNewNVIDIAFireworks AITogether AI○ sign inBatch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.Open full answer →