Practice tests · 30 items

Attention Kernels practice test

Online softmax and its proof, FlashAttention 1 to 3 and what each generation changed, split-KV decoding, paged attention kernels, MLA absorption, sliding windows, and why IO, not FLOPs, is the cost model. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test

Topic

Difficulty

Questions

10 in this pool · about 7 min

Reveal answers

Sample items with answers

easy · sample

Attention logits in a row reach 100. Why does every softmax kernel subtract the row maximum before exponentiating?

It makes the result more accurate, since softmax without it is only ever an approximation

It lets the kernel skip the normalizing division, since the max row term becomes 1

It turns the softmax into a sparse operation, since most shifted terms round to zero

✓exp(100) overflows FP32 (max near e^88.7), and the shift leaves the result unchanged

FP32 tops out near 3.4e38, which is e^88.7, so exp(100) is infinity and the row becomes NaN after division. Softmax is invariant to subtracting any constant from every logit, because the factor e^-c cancels between numerator and denominator. Subtracting the max puts every exponent at or below 0, so terms land in (0, 1]. The normalizing division is still needed, and the result is exact with or without the shift when no overflow happens. FP16 is worse: it overflows past e^11.1.

easy · sample

What does FlashAttention change relative to a standard attention implementation?

It approximates softmax with a low-rank kernel, cutting FLOPs from quadratic to linear in T

✓It never writes the T x S score matrix to HBM, computing the same result tile by tile

It drops attention to distant tokens, keeping only a fixed window per query

It moves attention from tensor cores to CUDA cores, which handle softmax faster

Standard attention writes S = QK^T to HBM, reads it back for softmax, writes P, and reads P again for PV, so HBM traffic grows with T x S. FlashAttention tiles Q, K and V into shared memory and uses the online softmax recurrence so the score tiles never leave the SM. FLOPs are the same (slightly more, from rescaling) and the output is exact up to floating-point order. Low-rank and windowed schemes change the math; FlashAttention changes only where intermediate values live.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: