Mistral AI Inference Engineer interview questions

Active parameters describe one token. A decode step serves a batch, and the batch decides how many experts get pulled through HBM. The answer computes that number and shows where the MoE cost advantage actually comes from.

Half the layers capped at a 128-token window sounds like a big saving at 128K. The arithmetic says the saving is set by the layer ratio, not the window, and the allocator decides whether you get it at all.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

With T new queries over S cached keys, the obvious lower-triangular mask is wrong and still runs without an error. The answer derives the right one from absolute positions and names the library default that gets it wrong.

GQA shrinks the cache by the group size, but a kernel that loops over query heads reads it back group-size times anyway. The answer packs the group into the matrix dimension and shows what that does to intensity and tensor-core utilization.

A 4K window over 32K tokens keeps one eighth of the keys, and the speedup over causal attention is not 8x. The answer counts tiles exactly, classifies which need masking, and explains where the rest of the gap goes.

A Python loop over 256 experts is correct and slow for three separate reasons, only one of which is launch count. The answer builds the expert-sorted layout the fused kernel runs on and counts what it costs in padding.

Local-inference formats spend every bit where it buys the most accuracy, and datacenter formats spend them uniformly. The answer works out what a Q4_K_M file actually costs per weight, and explains why the trade that wins on one consumer GPU loses at batch 64.

Calibration data reaches each expert only through the router, so the experts that matter for rare domains see almost none of it. The answer counts the tokens, shows the point where a Hessian-based quantizer becomes worse than plain rounding, and builds a recipe that does not depend on luck.

Expert parallelism runs at the speed of the busiest GPU, and the router decides which GPU that is. The answer finds the batch size at which skew starts to cost compute, measures what placement and replication recover, and shows why a placement fit to yesterday's traffic decays.

One engine is built to run a model anywhere with one user; the other is built to share a datacenter GPU across hundreds. The answer turns that into memory-bandwidth arithmetic and finds the cases where the laptop wins.

Perplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.

The attention math is ten lines. The grade comes from where the new keys land, which query head reads which KV head, and the causal mask when a chunk of queries sits at the end of a longer cache.

RoPE is a rotation per pair of dimensions. The bugs live in which dimensions pair up, which position each row of a ragged batch is at, what left padding does to position ids, and the precision of the angle at position 100,000.

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

Per token, moving KV over PCIe is several times cheaper than recomputing it, yet vLLM recomputes. The answer derives the break-even bandwidth, shows how small copies fall below it, and names what makes recompute nearly free.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.