The inference engineer interview process

Inference loops are assembled from a small number of round types, in proportions that depend on the team. A kernel team weights the performance rounds; a serving platform team weights system design and debugging; a lab's inference team usually tests all of them. Company pages on this site describe what public reporting supports for each employer and say so when the evidence is thin. Treat any loop description, including ours, as a starting point to confirm with your recruiter.

Expect to write, in Python or PyTorch, a component of a serving engine: grouped-query attention with an incremental KV cache, numerically stable attention over blocks, a paged block allocator with copy-on-write, a continuous-batching scheduler, a lossless speculative-decoding verifier or a top-p sampler. The code is graded on correctness first, then on the invariants and edge cases: a sequence that ends mid-block, a causal mask when the query is shorter than the key, a double free.

What it measures: whether you know the mechanism well enough to build it, not whether you remember an API. Stating the invariants before writing code is the single habit that most separates strong candidates.

Often folded into other rounds rather than run alone. Produce, without a calculator and with units, the KV cache bytes per token for a given config, the decode latency floor for a model on given GPUs, the batch size where decode turns compute-bound, the expected tokens per step of speculative decoding, or the GPU count for a traffic profile.

What it measures: whether your mental model of the hardware is quantitative. Interviewers listen for the regime (bandwidth-bound or compute-bound), for dense rather than sparse peak figures, and for a sanity check at the end.

Design an LLM API, a multi-tenant serving platform, a batched inference service on a fixed GPU pool, or a fleet that serves several models. The strong version starts with arithmetic: tokens per second in prefill and in decode, KV capacity per replica, replicas at a utilization ceiling, cost per million tokens. Only then do router, scheduler and autoscaler choices follow.

What it measures: whether the design is derived from the load or drawn from a template. Constraint changes mid-round (traffic doubles, contexts grow to 128K, a GPU generation changes) are normal.

For kernel and performance roles: explain FlashAttention's online softmax and its IO cost, optimize a GEMV or reduction, reason about coalescing, occupancy and bytes in flight, read a profiler report, or walk through what changes on a new GPU generation. Some teams set a take-home kernel optimization with a correctness and speed bar.

What it measures: whether you reason from the memory hierarchy and the roofline, and whether your performance claims come with a measurement plan.

A production symptom with partial evidence: p99 time to first token tripled after a deploy, an engine OOMs at peak, outputs differ between runs, one replica is slower than the rest, a quantized model regressed on one task. The interviewer reveals metrics as you ask for them.

What it measures: hypothesis ranking and evidence discipline. Strong candidates decompose latency into queue, prefill and decode time before guessing, and say which metric would distinguish their top two hypotheses.

The behavioral round, usually about a performance regression you owned, a trade-off between latency and cost you had to argue, working with researchers whose model was hard to serve, or a rollout you stopped. Specific numbers carry more weight here than in most behavioral rounds, because the job is measured in them.