decode

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

Batching rescues the weight GEMMs at decode but never the attention kernel, and one head-count ratio explains why. The answer derives it, then shows how MLA moves attention to within a hair of the H100 ridge.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

The datasheet ratios are close to each other, so the lazy answer is one speedup for everything. The answer splits it into three regimes, and the biggest win turns out to come from capacity, not bandwidth.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

Step time is a straight line in batch size, so throughput is a hyperbola with a ceiling. The answer finds the knee in closed form and shows why the ceiling usually comes from the KV cache, not the tensor cores.

Joules per token is power over throughput, and the power term is the one candidates obsess over. The answer shows why the denominator moves the result by about 50x while the numerator is pinned under a datasheet cap.

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

Every weight GEMM ties, attention sits at a constant the batch cannot move, and the ranking flips at batch 8. The answer then separates what fusion can fix from what only the model architecture or the batch can.

Absorption is one line of matrix associativity. The part interviewers score is the bill: attention gets more expensive per cached token, and the kernel lands in a roofline regime GQA never visits.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.

Batch-1 decode at long context gives the GPU eight thread blocks and 132 SMs. The answer counts the waves, picks the split count from a wave-efficiency rule rather than a guess, and derives the merge that makes the result exact.

The indirection adds 4 bytes per 64 KiB of KV, so the byte count barely moves. The answer shows where the real cost hides: dependent loads, tiles that span several pages, and copy engines that want one contiguous box.

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

Two GEMV kernels do the same arithmetic on the same matrix and differ only in which thread reads which element. One streams at HBM speed; the other turns every warp load into 32 separate transactions.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

A 4-bit weight is only cheaper if turning it back into FP16 costs less than the bytes it saved. The answer counts the instruction budget per weight at full HBM bandwidth and shows the bit trick that makes the conversion fit.

Every kernel in the step can run near peak bandwidth while the step itself runs at two thirds of it. The answer reads the three rows of the timeline that tell you why, and turns the gaps into milliseconds.

CUDA lets you reserve part of L2 for data you choose, and the KV cache is re-read every step. The answer measures what fits against what a decode step streams, finds the one case where pinning pays, and the kernel change that pays more.

A token still has to pass through every layer in order, so splitting the layers across nodes cannot make one token faster. The answer shows what pipelining buys instead, prices the bubble, and finds the all-reduce latency at which it beats cross-node tensor parallelism.

A ring all-reduce is bandwidth-optimal and takes fourteen synchronized steps on eight GPUs. At 16 KB per call, the steps are the cost. The answer builds the latency-bandwidth model, finds the crossover size, and checks it against the threshold vLLM ships.

Inside one batch there is nothing to overlap: every layer waits for its all-to-all. Split the batch in two and one half computes while the other communicates. The answer models the gain, finds the batch size where splitting makes things slower, and checks it against a shipped default.

A graph freezes shapes, so engines capture dozens of them and round every batch up. The answer prices that rounding in the regime where decode lives and finds the real costs somewhere else.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

Voice moves the latency target from the first token to the first audible clause, and the largest item in the budget is not a model at all. The answer allocates 300 ms line by line, then sizes the LLM tier from it.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

RoPE is a rotation per pair of dimensions. The bugs live in which dimensions pair up, which position each row of a ragged batch is at, what left padding does to position ids, and the precision of the angle at position 100,000.