Easy inference interview questions

Two FLOPs per parameter is the number everyone quotes. The answer interviewers want rebuilds it from the config, finds the parameters that do no arithmetic, and names the context length where the rule breaks.

The bound is a one-line argument about where a sequence can be partly full. The block size is a three-way trade between that waste, kernel efficiency and how finely the prefix cache can match.

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.

Two GEMV kernels do the same arithmetic on the same matrix and differ only in which thread reads which element. One streams at HBM speed; the other turns every warp load into 32 separate transactions.

Both formats spend the same eight bits. The answer derives where each one's largest and smallest values come from, then explains why inference almost always picks the narrower of the two.

Three knobs, one distribution, and an ordering question that most candidates have never thought about. The answer works one six-token example through every knob and shows the order changing which tokens can be sampled.

At temperature zero, verification is a prefix match, and the whole implementation fits in a dozen vectorized lines. Interviewers grade the three details around it: the bonus token, the KV slots to roll back, and why the output can still differ from plain greedy.

Two matrix multiplies in a row, and the order of the splits decides whether you communicate once or twice per block. The answer places every split for a 70B layer and prices what the remaining all-reduces send per token.

Three flags, three different budgets, and one startup check that fails a 70B deployment on two H100s with a message most people misread. The answer walks the memory profile in the order vLLM runs it.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

The attention math is ten lines. The grade comes from where the new keys land, which query head reads which KV head, and the causal mask when a chunk of queries sits at the end of a longer cache.

Everyone can sort and cumsum. The grader feeds you tied logits, p = 1 on a 128K vocabulary and a temperature of zero, and checks whether your kept set depends on the order of the sort.