Inference Engineer Interview Questions

196 inference engineer interview questions with worked answers, organized by the twelve topics an inference loop tests. Every answer carries its arithmetic as a chain you can recompute, names the hardware and precision behind each number, and ends with how a panel scores it. The topics follow the path of a request, from the router and the engine loop down to the attention kernel and the GEMM.

196 questions12 topics35 companiesupdated 2026

Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

What inference engineer interviews test

Inference loops are built from five kinds of round. The proportions vary by team, but each one expects numbers, not adjectives.

Implement-from-scratch coding
Attention with a KV cache, a paged block allocator, a continuous-batching scheduler or a speculative verifier, graded on invariants and edge cases.
Inference math
KV bytes per token, decode latency floors, crossover batch sizes and fleet sizing, produced quickly, with units and a sanity check.
Serving system design
An LLM API or multi-tenant platform sized from the load: tokens per second per phase, replicas, routing, autoscaling and cost per token.
Kernels and GPU performance
FlashAttention's online softmax, GEMV bandwidth, occupancy and bytes in flight, tensor-core generations and profiler evidence.
Debugging from metrics
A TTFT regression, an OOM at peak or nondeterministic outputs, decomposed into queue, prefill and decode time before any guess.

Inference Physics & Roofline Interview Questions

Bytes and FLOPs per token, why batch-1 decode reads every weight once per step, the ridge point and the batch size that crosses it, MBU versus MFU, and the millisecond latency floors you must produce before anyone draws a box.

See all 15 Inference Physics & Roofline questions with answers →

KV Cache Interview Questions

The KV algebra across MHA, GQA, MQA and MLA, block tables and fragmentation, hash and radix prefix reuse, eviction, FP8 KV, CPU and NVMe offload, and the gpu_memory_utilization budget that decides how many sequences actually fit.

See all 16 KV Cache & GPU Memory questions with answers →

Continuous Batching & Scheduling Interview Questions

Static, dynamic and iteration-level batching, chunked prefill and the token budget, preemption by swap or recompute, priority and fairness, queueing arithmetic, and the goodput a scheduler is really optimising.

See all 15 Batching, Scheduling & Admission questions with answers →

FlashAttention & Attention Kernel Interview Questions

Online softmax and its proof, FlashAttention 1 to 3 and what each generation changed, split-KV decoding, paged attention kernels, MLA absorption, sliding windows, and why IO, not FLOPs, is the cost model.

See all 16 Attention Kernels questions with answers →

GPU Performance Interview Questions for Inference

The memory hierarchy in numbers, coalescing and bank conflicts, occupancy, Hopper TMA and WGMMA, Blackwell tensor memory, decode GEMV versus prefill GEMM, fused MoE and dequant kernels, CUDA graphs, and reading Nsight like an interviewer.

See all 15 CUDA, Triton & GPU Performance questions with answers →

LLM Quantization Interview Questions

FP8 E4M3 and E5M2 scaling, INT4 weight-only kernels, GPTQ, AWQ, SmoothQuant, EXL2 and GGUF k-quants, MXFP4 and NVFP4 block scales, KV quantization, accuracy evaluation beyond perplexity, and batch-invariant determinism.

See all 15 Quantization & Numerics questions with answers →

Speculative Decoding Interview Questions

The draft and verify algebra with its losslessness proof, acceptance rate versus draft length, Medusa, EAGLE and multi-token prediction heads, tree verification, why speculation loses at high batch, sampling kernels, and grammar-constrained output.

See all 15 Speculative & Constrained Decoding questions with answers →

Distributed & Disaggregated Inference Interview Questions

Tensor, pipeline, expert and data-parallel attention, the all-reduce bytes per layer, MoE dispatch and combine all-to-all, prefill and decode disaggregation, KV transfer over NVLink and RDMA, and rack-scale NVLink domains.

See all 15 Distributed & Disaggregated Inference questions with answers →

vLLM, SGLang & TensorRT-LLM Interview Questions

vLLM V1, SGLang, TensorRT-LLM and llama.cpp from the inside: the engine core loop, schedulers, KV managers, model runners, CUDA graph capture, detokenisation and streaming, and the configuration knobs whose defaults you must be able to defend.

See all 16 Serving Engine Internals questions with answers →

LLM Serving System Design Interview Questions

Design an LLM API on a fixed GPU pool: prefix and KV-aware routing, autoscaling on the right signal, multi-LoRA and multi-model serving, tenant isolation, cold starts and weight distribution, and cost per million tokens under an SLO.

See all 16 Inference System Design questions with answers →

Inference Performance Debugging Interview Questions

p99 TTFT regressions with flat utilisation, OOMs under continuous batching, numerical divergence after an engine upgrade, nondeterministic outputs, slow replicas, and benchmark methodology that does not lie about warm-up, clocks or prompt mix.

See all 21 Performance Debugging & Benchmarking questions with answers →

Inference Coding Interview Questions

The coding round of an inference loop: grouped-query attention with a cache, online-softmax attention, a paged block allocator, a continuous batching scheduler, a lossless speculative verifier, top-p sampling, RoPE, INT4 packing and Triton kernels.

See all 21 Implement-from-Scratch Coding questions with answers →

Inference interview questions by company

Each company weights the rounds differently. These pages say what public evidence supports about each loop, with a confidence level, and say so when the evidence is thin.

Before the interview: the role itself

What is an inference engineer?The engineer between a trained model and the token a user sees.Inference engineer interviewsSix round types, and the signal each one is scored on.Inference engineer salaryEmployer-posted base bands by company and level, labelled by source.Who hires inference engineersLabs, providers, silicon vendors, big tech and trading firms, with careers links.Inference engineer skills roadmapWhat to learn first, and why the order matters.Inference vs ML engineerWhere the role ends and ML, platform and kernel roles begin.Readiness assessmentA free diagnostic across the twelve topics an inference loop tests, with a study plan.

Inference engineer interviews: FAQ

What is asked in an inference engineer interview?

Implement-from-scratch coding of serving components (attention with a KV cache, a paged block allocator, a scheduler, a speculative verifier), inference math such as KV bytes per token and decode latency floors, serving system design sized from a traffic profile, kernel and GPU performance deep dives, and a debugging case built from production metrics.

How is an inference interview different from an ML engineer interview?

An ML engineer loop tests modelling: data, objectives and evaluation. An inference loop starts after the model is trained and tests how it is served: memory, batching and scheduling, kernels, number formats, parallelism and cost per token, usually with arithmetic expected in every round.

What math do inference engineer interviews expect?

Back-of-envelope numbers produced quickly and with units: KV cache bytes per token for a given config, the decode latency floor from weight bytes and memory bandwidth, the batch size where decode turns compute-bound, expected tokens per step of speculative decoding, and GPU counts and cost for a traffic profile.

Do I need to know CUDA for an inference engineer interview?

For kernel and performance roles, yes: expect CUDA or Triton, the memory hierarchy, coalescing, occupancy and profiler evidence. For engine and fleet roles, expect to reason about kernels and read profiles rather than write them from scratch.

How should I prepare for an LLM inference interview?

Start with the cost model (the roofline, prefill versus decode), then KV memory and scheduling, then one serving engine read from its current source, then numerics and kernels, then parallelism and fleet design. Practise each answer out loud with the arithmetic, and check it against the scoring lines.

Every inference interview question, worked to the number

196 questions across 12 topics, ordered easy to expert, each with the derivation, the follow-ups interviewers hold back and the scoring lines. The first questions in every topic are free.

Start with the free questions →Unlock all answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

A serving team at a lab and a kernel team at a chip vendor run different loops, and both change between hiring cycles. Ask your recruiter which rounds your loop includes before you plan around this list.