What is an inference engineer?

An inference engineer makes a trained model cheap, fast and reliable to run. The research team hands over weights and a config file; the inference engineer decides how those weights are split across GPUs, which number format they are stored in, how requests are batched and scheduled, how the KV cache is managed, which kernels run each layer, and how the fleet scales, then owns the latency and cost of every token that results.

The job is measured in a small set of numbers: time to first token, time per output token, requests that meet both targets (goodput), and dollars per million tokens. Almost every decision the role makes moves one of those at the expense of another, which is why interviews for it are full of arithmetic.

The work splits into layers that mirror the path a request takes. At the bottom are kernels: attention, matrix multiplication, sampling and dequantization code in CUDA or Triton, tuned to a specific GPU generation. Above them sits numerics: deciding when FP8, INT4 or a 4-bit block-scaled format keeps quality acceptable and where the scales live. Above that is the engine: the scheduler that decides which tokens to run each step, the KV cache manager that allocates memory in blocks and reuses shared prefixes, and the model runner that captures the decode step as a CUDA graph.

Above the engine is distribution, splitting a model too large for one GPU with tensor, pipeline or expert parallelism and pricing the communication each split adds. At the top is the fleet: routing requests to replicas that already hold their prefix, autoscaling on queued tokens rather than GPU utilization, isolating tenants, and turning a GPU-hour price into a cost per token.

Most inference engineers specialize in one or two of these layers and are expected to reason fluently about all of them. A kernel engineer still needs to know why the scheduler sends a batch of 256 decode tokens; a fleet engineer still needs to know why a decode step is priced in bytes.

The one fact everything else rests on: generating a token at small batch sizes reads every weight in the model once and does about two floating-point operations with each. That is roughly one operation per byte in BF16, against an H100 that can do about 295 operations in the time it moves one byte from memory. Decode is therefore limited by memory bandwidth, not arithmetic, and most of the job is either reducing bytes (quantization, grouped-query attention, smaller KV caches) or spending the idle arithmetic usefully (batching more sequences, speculative decoding).

Prefill, processing the prompt, is the opposite: thousands of tokens share each weight read, so it is limited by arithmetic. A serving system is two workloads with opposite bottlenecks sharing one GPU, and much of inference engineering is managing that tension.

Titles vary more than the work does. Frontier labs post Software Engineer or Member of Technical Staff roles on an inference or model serving team. Accelerator companies post deep learning performance engineer, inference optimization engineer and kernel engineer roles. Inference providers and AI platform teams post ML systems engineer, AI infrastructure engineer or serving engineer roles. Read the responsibilities: if they name a serving engine, latency or throughput targets, kernels, quantization or KV cache work, it is this job.

It is distinct from machine learning engineering, which builds and evaluates models, and from broad ML platform work, which runs training pipelines, clusters and data systems. The guide on how the role differs from adjacent roles covers where the lines fall.