← 🔦 Attention Kernels
Advanced
Split-KV Decoding (Flash-Decoding)
In decode there is one query row per sequence, so a FlashAttention kernel parallelized over batch and heads launches too few thread blocks to pull full HBM bandwidth. Split-KV divides each sequence's keys across blocks and merges the partial results exactly with their log-sum-exp, which decides whether long-context decode reaches bandwidth.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Inference Physics & RooflineWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?→KV Cache & GPU MemoryDerive the KV cache footprint for MHA, GQA and MLA, then tell me how many 8K-token conversations fit on four H100s.→Inference System DesignSize the fleet for a 70B chat API at 200 req/s, 2K tokens in and 300 out, p99 TTFT under 1 s, TPOT under 50 ms. What is the token cost?→Inference Physics & RooflineDerive the FLOPs per token for a dense transformer from its config. Where does the 2N rule stop being accurate?→Attention KernelsWrite the causal mask for a decode step with a KV cache, where the query is shorter than the keys. Which corner does it align to?→CUDA, Triton & GPU PerformanceIn a decode GEMV over a row-major weight matrix, which thread-to-element mapping coalesces, and why does the transposed one not?→