Advanced

FlashAttention 1 to 4: What Each Generation Changed

Every FlashAttention generation computes the same exact attention and attacks a different bottleneck: HBM traffic in FA1, poor work partitioning in FA2, Hopper's asynchronous tensor cores in FA3, and Blackwell's slow exponential units in FA4. Knowing which bottleneck each one removed tells you which kernel a given GPU should run.

Get full Premium access · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

RELATED CONCEPTS

The Online Softmax Recurrence Split-KV Decoding (Flash-Decoding)The GPU Memory Hierarchy in Numbers Bytes in Flight: Little's Law for GPU Memory Block-Scaled FP4: MXFP4 and NVFP4 The Roofline Model for LLM Inference

PRACTICE THIS IN REAL QUESTIONS

Attention KernelsDerive online softmax, prove it is exact, and explain why it makes FlashAttention fast without reducing FLOPs.→Quantization & NumericsAWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.→Quantization & NumericsFP8 comes in E4M3 and E5M2. Give the range and precision of each, and say which tensors use which in inference.→Attention KernelsImplement online softmax for a whole tile of queries, not one query, and write the rescaling step in matrix form.→CUDA, Triton & GPU PerformanceA kernel uses 128 registers per thread and 48 KB of shared memory per block. How many warps are active per SM on an H100?→Quantization & NumericsPer-tensor, per-channel, per-group, per-block: where does each quantization scale live, and what does it cost in bytes and in the kernel?→