Inference engineer skills: a roadmap in the order they compound
The skills an inference engineer needs, in the order each one makes the next easier: the roofline, the transformer forward pass in bytes, KV memory, batching and scheduling, a serving engine's internals, numerics, kernels, parallelism and fleet design.
7 MIN READ · UPDATED 16 OCTOBER 2026
PRACTICE THIS:Inference Physics & Roofline ·KV Cache & GPU Memory ·CUDA, Triton & GPU Performance
Start with the cost model, not the tools
Every later skill is easier once you can price an operation in bytes and FLOPs and place it on a roofline. Learn to compute arithmetic intensity, the ridge point of a GPU from its datasheet (dense figures, not sparse), and what it means to be bandwidth-bound or compute-bound. Then apply it to the two phases of generation until the asymmetry between prefill and decode is automatic.
Then the transformer, counted
Read a model's config file and derive its parameter count, the bytes it streams per decode step, and the KV cache it stores per token for multi-head, grouped-query and multi-head latent attention. A candidate who can do this for a model released last week, without being told the answer, is already ahead of most of the field.
Then memory and scheduling
Learn why paged KV memory replaced contiguous reservation, how block tables, reference counts and copy-on-write work, and how prefix caching identifies shared blocks. Then learn iteration-level batching, chunked prefill and the token budget, preemption, and the queueing behaviour that turns modest utilization increases into large tail-latency increases. Build a small allocator and scheduler yourself; it is the fastest way to make the ideas stick, and interviews ask for exactly that code.
Then one engine, read from source
Pick vLLM or SGLang and follow a request through it in the current source, not a blog post from two releases ago: where tokenization happens, how the scheduler builds a step, how the KV manager allocates, where CUDA graphs are captured. Note which defaults matter and which version you read, because both change quickly.
Then numerics and kernels
Learn what FP8 E4M3 and E5M2 can represent, how weight-only INT4 differs from W8A8, and how block-scaled 4-bit formats store their scales. Then learn enough CUDA or Triton to write a correct reduction, softmax and GEMV, explain coalescing and occupancy, and read a profiler report. The goal is not to out-write a kernel library; it is to know why a kernel is slow and what to ask for.
Finally parallelism and fleets
Tensor, pipeline and expert parallelism, the bytes and the call count each adds per token, and when prefill and decode deserve separate machines. Then fleet design: routing for prefix reuse, autoscaling signals, tenant isolation and cost per token. These are the staff-level rounds, and they are only convincing when the lower layers are solid.
Question topics that test what this guide covers:
FAQ
Roofline reasoning, transformer cost accounting from a config file, KV cache memory management, batching and scheduling, serving engine internals, number formats and quantization, enough CUDA or Triton to diagnose kernels, model parallelism, and fleet-level capacity and cost planning.