memory bandwidth

Decode reads every weight once per token. The answer turns that sentence into a millisecond floor, then accounts for the three things that keep a real server above it.

One division tells you whether a headline number is physics or marketing. The answer runs it, then works out which unstated condition would make the claim true and what to ask the vendor for.

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

MBU is one division, and that is why it is so easy to get wrong. The answer defines both operands precisely on a real server and then shows six ways a dashboard can report a number 30 points off.

A 60% MBU number is a symptom, not a diagnosis. The answer rebuilds the floor from bytes, then splits the missing 40% into buckets a profiler can confirm and a team can own.

The crossover falls out of one division, and the number it gives lines up exactly with something else on the GPU. Spotting that coincidence is what the interviewer is waiting for.

The textbook IO count says FlashAttention re-reads K and V once per query block, which at 8K is only 2x better than naive. The answer explains why the real HBM number is 65x lower, and which cache makes the difference.

A matrix-vector product has one job: keep enough bytes in flight. The answer counts how many that is per SM, then finds the three places a decode GEMV quietly drops below it.

Two GEMV kernels do the same arithmetic on the same matrix and differ only in which thread reads which element. One streams at HBM speed; the other turns every warp load into 32 separate transactions.

RMSNorm does almost no arithmetic, so its cost is bytes, and the eager PyTorch version moves nine times more of them than it needs to. The answer writes the one-pass kernel, tests it on awkward hidden sizes, and says where fusion stops paying.