long context

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

FP8 halves the cache and the attention bytes, and on short prompts it looks free. The answer explains why the scale matters less than people think for rounding and more than they think for clipping, and why the failures hide at 128K.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

The per-token formula gives a scary number for GQA and a reassuring one for MLA. Put both on real GPUs with tensor parallelism and the reassuring one is the request that does not fit.

Keys and values take the same bytes, yet their errors reach the output by different roads. Trace both roads and the scale layout for each tensor follows, along with the bookkeeping that makes per-channel keys awkward to serve.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

Fusing RoPE saves a read and a write of Q and K, which is real but small in prefill. The precision bug is not small: one tensor in the wrong dtype scrambles positional information by a few hundred tokens. The answer sizes both.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

At 1M tokens a 70B GQA model needs 320 GiB of KV per request and spends over 90% of prefill FLOPs in attention. The answer lays out a plan that fits on eight H100s, then shows which single change to the model breaks it and what replaces it.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Perplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.