split kv

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

At 1M tokens a 70B GQA model needs 320 GiB of KV per request and spends over 90% of prefill FLOPs in attention. The answer lays out a plan that fits on eight H100s, then shows which single change to the model breaks it and what replaces it.

Review a shape-by-shape timing fixture, catch a ragged-cache correctness failure, and calculate whether selective dispatch pays for itself. Includes a small reference you can run without a GPU.

At batch 1 a decode attention kernel gives each KV head one thread block and leaves most of the GPU idle. The answer splits the sequence, carries a log-sum-exp with every partial output, merges them exactly, and survives the split that has no keys at all.