numerics

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

FlashAttention does the same arithmetic as naive attention and is several times faster. The answer is a three-line recurrence and one IO count, and interviewers want both.

The one-query recurrence is a warm-up. A real kernel carries a vector of maxima and a vector of sums for a tile of rows, and the rescale becomes a diagonal matrix. The answer writes it, runs it, and shows the edge case that turns it into NaN.

Three quantization points sit inside FP8 attention, and they fail for different reasons: outlier channels in keys, a probability matrix that lives near zero, and accumulators. The answer simulates each one and says which fix addresses which.

Fusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

Lossless is a statement about one distribution, and quantization changes which one without raising any error. The answer separates the draft case from the target case, then shows two ways an FP8 deployment breaks exactness without touching the rejection rule.

Greedy decoding is deterministic on paper and the customer has two different answers in hand. The answer finds where the bits change, shows a measured example of it, and turns the finding into a promise the platform can keep.

Everyone can sort and cumsum. The grader feeds you tied logits, p = 1 on a 128K vocabulary and a temperature of zero, and checks whether your kept set depends on the order of the sort.

RoPE is a rotation per pair of dimensions. The bugs live in which dimensions pair up, which position each row of a ragged batch is at, what left padding does to position ids, and the precision of the angle at position 100,000.