LLM inference glossary
134 terms from the serving path, grouped by the interview round that tests them. A definition is where an answer starts, so most entries add the figure that makes the term concrete, with its part, precision or engine version, and the term it is most often mistaken for. Every figure is derived on the page the entry links to.
134 terms available.
Inference Physics & Roofline13 terms · questions
- Roofline model
A bound on how fast a kernel can run: the smaller of the hardware's peak compute and its memory bandwidth times the kernel's arithmetic intensity.
On an H100 SXM the BF16 ridge sits at 989 / 3.35 = 295 FLOP per byte; batch-1 decode is near 1, so it is bandwidth-bound by two orders of magnitude.
- Arithmetic intensityoperational intensity
FLOPs performed per byte moved to or from memory.
A BF16 weight GEMM at decode batch B does about B FLOPs per byte of weight read, which is why batching moves decode toward the ridge.
- Ridge point
The arithmetic intensity at which a kernel stops being bandwidth-bound and becomes compute-bound: peak FLOP/s divided by bandwidth.
295 FLOP per byte for dense BF16 on an H100 SXM.
- Prefill
The forward pass over a request's prompt, which computes the KV for every prompt token and produces the first output token.
Compute-bound: a 4,096-token prompt on Llama 3.3 70B at TP=2 on H100 SXM is 303 ms at peak, 0.074 ms per prompt token.
Not the same as Decode, which processes one token per sequence per step and is bound by bytes read.
- Decode
Generating one token per running sequence per step, which reads every weight once and every cached token of every sequence.
The same Llama 3.3 70B deployment reads 71.2 GB per GPU per step, 21.3 ms: one output token costs about 300 prompt tokens' time at batch 1.
- MFUmodel FLOPs utilization
The model's useful FLOPs per second divided by the hardware's dense peak.
Not the same as GPU utilization, which reports whether any kernel is running and reads near 100% at almost any load.
- MBUmodel bandwidth utilization
Bytes a decode step must read (weights plus KV) divided by step time, as a fraction of peak memory bandwidth.
The decode counterpart of MFU: a step at 70% MBU is close to as fast as the memory allows.
- Bandwidth floor
The shortest a decode step can be: the bytes it must read divided by memory bandwidth.
Llama 3.3 70B in FP8 on one TPU7x chip at 7,380 GB/s: 70.55 GB / 7.38 TB/s = 9.6 ms per step.
- TTFTtime to first token
Time from a request's arrival to its first output token: queueing, prefill and the first sampling step.
- TPOTtime per output token
A request's decode time divided by its output tokens after the first, averaged per request.
Not the same as ITL, the gap between consecutive tokens, measured per gap; a stream can have a good TPOT and one long freeze.
- ITLinter-token latency
The time between two consecutive tokens of one stream, measured per gap.
- Goodput
Completed requests per second that met every SLO, as opposed to all completions per second.
- Dense versus sparse TFLOP/s
Vendor peaks marked "with sparsity" assume 2:4 structured sparsity and are twice the dense figure; a dense model runs at the dense number.
The L40S lists 362 dense BF16 TFLOP/s and 733 with sparsity.
KV Cache & GPU Memory15 terms · questions
- KV cache
Each layer's keys and values for every token already processed, kept so decode does not recompute them.
Per token it costs 2 x layers x KV heads x head dim x bytes: 128 KiB for Llama 3.1 8B and 320 KiB for Llama 3.3 70B in BF16.
- Grouped-query attentionGQA
Several query heads share one key-value head, which divides the KV cache by the group size.
Llama 3.3 70B has 64 query heads and 8 KV heads, 8x less KV than full multi-head attention.
- Multi-query attentionMQA
The limit of GQA: one key-value head shared by every query head.
- Multi-head latent attentionMLA
Keys and values are compressed into one shared latent per token, which is all that is cached, plus a small RoPE key that cannot be absorbed into it.
In DeepSeek V3 each layer stores 576 numbers per token, the 512-dimension latent plus the 64-dimension RoPE key: 70,272 B per token over 61 layers in BF16.
Not the same as Caching separate key and value latents, which would double the figure.
- Paged KV cachePagedAttention
The KV pool is cut into equal blocks of tokens and each sequence keeps a table of the blocks it owns, so its cache need not be contiguous and its waste is bounded by its last, partly filled block.
vLLM v0.32.0 uses 16-token blocks on CUDA by default.
- Block table
A sequence's list mapping its logical KV blocks to physical blocks in the pool.
- Prefix cachingautomatic prefix caching · APC
Reusing the KV computed for a prompt prefix that an earlier request already sent.
vLLM hashes each full 16-token block together with its parent's hash, so a hit on block i proves blocks 0 to i all match.
- Radix cacheRadixAttention
SGLang's prefix cache: a tree whose edges are token spans, which shares prefixes structurally and evicts only leaves.
- gpu_memory_utilization
vLLM's fraction of the CUDA-reported device memory the engine may use; the KV pool is what is left after weights, activations and other allocations.
0.92 by default in v0.32.0, applied to 79.18 GiB on an 80 GB H100, not to the 80 on the label.
- mem_fraction_static
SGLang's static share of device memory for weights and the KV pool, computed at startup when not set.
- Preemption
Freeing a running request's KV when the pool runs out, then recomputing or reloading it later.
Under its default FCFS policy, vLLM v0.32.0 preempts the newest running request and puts it at the front of the waiting queue.
Not the same as Retraction, SGLang's version, which picks the request with the fewest outputs.
- KV offload
Keeping KV in host memory or on SSD and fetching it back instead of recomputing the prefix.
It pays when a token's KV moves faster than it can be recomputed: an 8K Llama 3.1 8B prefix is 21 ms over PCIe Gen5 at an assumed 50 GB/s against about 251 ms of recompute.
- FP8 KV cache
Storing keys and values in an 8-bit float format with per-tensor or finer scales, halving KV bytes.
Without calibrated scales vLLM uses 1.0 and warns once at startup; check the warning before trusting the cache.
- Sliding-window attentionSWA
Layers that attend only to the last W tokens, so their KV per sequence stops growing at W.
- KV cache usagekv_cache_usage_perc
The share of the pool's blocks held by running requests.
Batching, Scheduling & Admission12 terms · questions
- Continuous batchingiteration-level scheduling · in-flight batching
Requests join and leave the running batch at every step instead of waiting for a whole batch to finish.
- Chunked prefill
Splitting a long prompt into chunks that fit a per-step token budget, so one prompt cannot stall decodes for its whole length.
- Token budgetmax_num_batched_tokens
The most tokens one engine step may process, prefill and decode together in vLLM.
8,192 for vLLM v0.32.0's OpenAI server on an 80 GB H100; 16,384 on parts with at least 160 GiB.
- max_num_seqs
vLLM's cap on running sequences.
1,024 for the OpenAI server on an H100; on a PCIe or memory-tight node it is often the setting that stops preemption.
- Admission control
The rule that decides when a waiting request may start holding KV.
TensorRT-LLM reserves prompt plus max_tokens, vLLM checks that the prompt fits, and SGLang books a forecast of the remaining output, so one pool admits a different number of requests in each.
- new_token_ratio
SGLang's forecast of how much of a running request's remaining max_tokens it will actually use.
- Retraction
SGLang returning running requests to the queue when the next decode token does not fit, fewest outputs first.
- Head-of-line blocking
A request at the front of the queue that cannot run holds back the requests behind it.
- LPM schedulinglongest prefix match
An SGLang policy that orders the waiting queue by cached prefix length to raise the hit rate.
- Mixed chunk--enable-mixed-chunk
SGLang's option to run decodes inside prefill batches, as vLLM does, instead of in separate passes.
Off by default in v0.5.22; it shortens the inter-token tail that separate prefill passes create.
- Load shedding
Rejecting requests at admission, before any GPU work, when they would miss their latency target anyway.
- Little's law
Items in a system equal arrival rate times time in the system, L = λW, for any stable queue.
40 requests/s with a 0.5 s mean queue time means about 20 requests waiting; a waiting gauge that disagrees is wrong.
Attention Kernels10 terms · questions
- FlashAttention
Exact attention computed in tiles held on chip, with an online softmax, so the full score matrix is never written to memory.
- Online softmax
Computing softmax in one pass by carrying the running maximum, the running sum and the weighted output, rescaling both when the maximum rises.
- Split-KV decodingFlash-Decoding
Splitting one sequence's KV across thread blocks during decode and merging the partial results with their log-sum-exp.
Without it a Llama 3.1 8B decode launches 8 blocks on a GPU with 132 SMs.
- Log-sum-exp mergeLSE merge
Combining partial softmax results computed over disjoint key sets into the exact full result.
- RoPErotary position embedding
Encoding position by rotating each pair of query and key dimensions by an angle proportional to the position, so scores depend on relative position.
Llama 3.1 uses base 500,000 over 128-dimensional heads.
- RoPE scalingllama3 scaling · YaRN · NTK scaling
Reshaping RoPE frequencies so a model reads positions beyond its pretraining length.
Llama 3.1 ships llama3 scaling with factor 8 over an original 8,192 positions, and a max_position_embeddings of 131,072.
- Attention sink
An extra logit in the softmax denominator with no value vector, which lets a head attend to nothing.
gpt-oss learns one sink logit per attention head.
- Sparse attentionDSA · top-k attention
Each query attends to a selected subset of cached tokens, chosen by a cheap scorer, instead of all of them.
DeepSeek-V3.2 selects 2,048 tokens per query; it cuts bytes read per step, not the resident cache.
- Attention backend
The kernel library an engine selects for attention, chosen per GPU generation from a priority list filtered by the configuration.
- Cascade attention
A kernel that reads a prefix shared by many sequences once per step and merges it with each sequence's own suffix.
Removed from vLLM in v0.32.0; without it, every sequence reads the shared prefix's KV itself.
CUDA, Triton & GPU Performance12 terms · questions
- Streaming multiprocessorSM
The GPU's unit of execution: a thread block runs on one SM, with its registers, shared memory and tensor cores.
An H100 SXM has 132.
- HBMhigh-bandwidth memory
Stacked DRAM beside the GPU die; its bandwidth sets the decode floor.
80 GB at 3.35 TB/s on an H100 SXM.
- Tensor core
Matrix-multiply units inside each SM that provide the dense peak figures for 16-bit and lower formats.
- Occupancy
Resident warps per SM as a share of the maximum, limited by registers, shared memory and block size.
Not the same as Utilization: high occupancy helps hide latency, but a kernel can be fast at low occupancy.
- CUDA graph
A recorded sequence of kernels replayed with one launch; it freezes shapes and addresses, so engines capture one per batch size.
vLLM v0.32.0 captures 51 sizes up to 512 on an H100 by default.
- Kernel launch overhead
Host time spent issuing each kernel; it caps a step when the GPU's work per step is shorter.
About 390 launches at an assumed 5 µs is 1.95 ms, longer than a 1.20 ms TP=4 decode step for Llama 3.1 8B.
- Kernel fusion
Combining consecutive operations into one kernel so intermediate results stay on chip.
- Grouped GEMM
One kernel launch that runs many independent matrix multiplies, such as every MoE expert's, from a list of tiles.
- Triton (language)
A Python-embedded language for writing GPU kernels at the level of tiles.
Not the same as NVIDIA Triton Inference Server, now Dynamo-Triton, which is a model server.
- Nsight Systems and Nsight Compute
NVIDIA's profilers: Systems shows the timeline of kernels, copies and host gaps; Compute measures one kernel in depth.
- Custom all-reduce
vLLM's low-latency all-reduce over peer-to-peer GPU memory, used instead of NCCL for small messages.
Quantization & Numerics12 terms · questions
- BF16bfloat16
A 16-bit float with FP32's 8 exponent bits and 7 stored mantissa bits: FP32's range at a unit roundoff of 2^-8.
- FP16half precision
A 16-bit float with 5 exponent and 10 mantissa bits: unit roundoff 2^-11 and a largest value of 65,504.
- FP8 (E4M3 and E5M2)
Two 8-bit float formats: E4M3 tops out at 448 with finer steps, E5M2 reaches 57,344 with coarser ones.
- NVFP4 and MXFP4
4-bit floats (E2M1) with block scales: NVFP4 adds an FP8 scale per 16 values and a per-tensor scale, MXFP4 a power-of-two scale per 32.
- Weight-only quantizationW4A16 · W8A16
Storing weights in fewer bits and dequantizing them inside the GEMM, while activations and math stay 16-bit.
It cuts the bytes decode reads but not prefill's compute.
- W8A8
Weights and activations both in 8 bits, so the GEMM runs on 8-bit tensor cores at twice the 16-bit peak.
FP8 W8A8 needs Hopper, Ada or Blackwell; the A100 has no FP8 tensor cores.
- Quantization scaleper-tensor · per-channel · per-block scale
The factor that maps a low-precision code back to a real value, shared by a tensor, a channel or a block of values.
Finer scales follow outliers more closely at the cost of more metadata and kernel support.
- AWQ and GPTQ
Post-training methods that choose 4-bit weight values and scales from a calibration set to limit output error.
- Marlin kernel
A mixed-precision GEMM kernel for 4-bit weights with 16-bit activations, designed to stay bandwidth-efficient at small batch.
- Unit roundoff
The largest relative error of rounding one value to a format, half its spacing at 1: 2^-8 for BF16, 2^-11 for FP16.
- Batch invariance
Giving the same bits for a request whatever else is in its batch, by fixing every reduction strategy.
Not the same as Agreement between two programs, such as a trainer and an engine, which batch invariance does not provide.
- Calibration
Running representative inputs through a model to choose quantization scales or values.
A scale fitted to the calibration set's maximum does not bound tomorrow's outlier.
Speculative & Constrained Decoding11 terms · questions
- Speculative decoding
A cheap drafter guesses the next few tokens and the large model scores every guess in a single forward pass; a rejection rule decides how many to keep, so the output distribution is exactly the large model's.
- Acceptance rate
The probability that the target accepts a drafted token.
With acceptance α and γ drafts a verification yields (1 - α^(γ+1)) / (1 - α) tokens: 2.94 at α = 0.7, γ = 5, not 1 + γα = 4.5.
- Bonus token
The extra token the verification pass samples when every draft is accepted.
- EAGLE, Medusa and MTP heads
Drafters built into or onto the target model, predicting future tokens from its hidden states.
- N-gram speculationprompt lookup
Drafting by copying the continuation of a matching n-gram from the prompt or earlier output, with no draft model.
- Temperature, top-k and top-p
Temperature divides the logits before softmax, flattening or sharpening the distribution. Top-k discards all but the k likeliest tokens; top-p cuts the sorted distribution once its cumulative mass reaches p.
- Constrained decodingstructured output · grammar-constrained decoding
Masking at each step every token that would break a grammar or schema.
- Jump-forward decoding
Appending text the grammar forces without running a model step for each token.
- Raw versus processed logprobs--logprobs-mode
Whether reported log-probabilities come before logit processors and sampling filters or after them.
vLLM v0.32.0 reports raw_logprobs by default, an engine-wide setting.
- Beam search
Keeping the k most likely partial sequences at each step instead of sampling one.
With paged KV, beams share blocks by reference until one writes into a shared partial block.
- Incremental detokenization
Turning token IDs into text as they stream, holding back bytes that do not yet form a whole UTF-8 character.
Distributed & Disaggregated Inference12 terms · questions
- Tensor parallelismTP
Splitting each layer's weight matrices across GPUs, which then combine partial results with two all-reduces per layer.
- Pipeline parallelismPP
Placing consecutive layers on different GPUs and passing activations between stages.
At decode each micro-batch re-reads its stage's weights, so it rarely beats tensor parallelism for throughput.
- Data parallelismDP · replicas
Running independent copies of the model, each serving its own requests.
- Expert parallelismEP
Placing a mixture-of-experts model's experts on different GPUs and sending each token to the GPUs that hold its experts.
- All-reduce
A collective that sums a tensor across GPUs and leaves the result on every one.
Llama 3.3 70B under tensor parallelism runs 160 per decode step, which is why per-call latency matters on PCIe.
- All-to-alldispatch and combine
A collective in which every GPU sends a different slice to every other GPU, used to route tokens to experts and back.
- NVLink
NVIDIA's GPU-to-GPU interconnect.
900 GB/s bidirectional per H100 SXM, 450 GB/s each way; PCIe Gen4 x16 is 64 GB/s bidirectional.
- Prefill-decode disaggregationPD disaggregation · disaggregated serving
Running prefill and decode on separate GPU pools and moving each request's KV between them.
It removes prefill's interference with decode and lets each phase pick its own parallelism; it does not make either phase cheaper.
- KV transfer
Moving a request's KV from the prefill pool to the decode pool, over NVLink or RDMA through libraries such as NIXL or Mooncake.
- DP attention--attn-dp-size
Data-parallel attention inside a tensor-parallel group: each rank holds whole sequences' KV while the rest of the layer stays sharded.
Used for MLA models whose latent cache would otherwise be replicated on every TP rank.
- Context parallelism
Sharding one sequence's tokens across GPUs so each attends over its share and the partial results merge.
- Routing replay
Recording the experts the rollout engine chose for each token so the trainer can route identically.
Serving Engine Internals12 terms · questions
- vLLM
An open-source serving engine built around paged KV, continuous batching and a central scheduler process.
- SGLang
An open-source serving engine built around a radix prefix cache, with one scheduler process per GPU.
- TensorRT-LLM
NVIDIA's serving engine and kernel library, now with a PyTorch backend next to its compiled engines.
- EngineCore
vLLM's per-data-parallel-rank process that owns the scheduler and the KV block pool and drives the workers.
- Model runner
The engine component that prepares a step's inputs, runs the forward pass and returns sampled tokens.
- Overlap schedulingasync scheduling
Preparing the next step on the CPU while the GPU runs the current one, so host work does not idle the GPU.
In SGLang it also means a pass's prompts reach the radix tree only after the next pass is formed.
- Prefix-aware routing
Sending each request to the replica most likely to hold its prefix, balanced against load.
- Multi-LoRA serving
Serving many low-rank adapters on one base model, batching requests for different adapters into one step.
- Sleep mode
vLLM's way to release GPU memory without exiting: level 1 offloads weights to host memory, level 2 discards them.
- KV connector
vLLM's interface for loading and saving KV outside the engine's own pool, used for offload tiers and disaggregation.
- generation_config
vLLM's setting that decides where default sampling parameters come from; "auto" reads the model repository's file.
- Chat template
The model's rule for turning a list of messages into one prompt string, including its special tokens.
Inference System Design8 terms · questions
- SLOservice-level objective
A target on a measured percentile, such as p99 TTFT of at most 500 ms, that the service commits to.
- Capacity planning in tokens
Sizing a fleet from prompt and output token rates priced in FLOPs and bytes, rather than from requests per second.
- Autoscaling
Adding and removing replicas from demand signals, with headroom for the time a new replica takes to start.
- Cold start
The time from a new replica's process start to serving: weight load, compilation, memory profiling, graph capture and readiness.
- Warm pool
Replicas kept loaded but idle, or asleep, so they can take traffic in seconds.
- Token quota
A per-tenant limit on tokens per interval, usually enforced as a token bucket charged at admission.
- Cost per million tokens
GPU cost apportioned to tokens by the resource each phase saturates: compute for input, bandwidth for output.
At an illustrative $2.50 per H100-hour, Llama 3.1 8B costs about $0.023 per million input and $0.089 per million output tokens.
- Cache retention guarantee
A promise that a prefix stays cached for a stated time, which needs reserved capacity rather than spare memory.
Performance Debugging & Benchmarking10 terms · questions
- Coordinated omission
A load generator that waits for slow responses before sending more, so it stops measuring exactly when the server stalls.
- Open-loop and closed-loop load
Open-loop clients send on a schedule regardless of responses; closed-loop clients keep a fixed number in flight.
- A/A test
Running the same configuration twice to measure the noise that any A/B difference must exceed.
- ABBA ordering
Interleaving two configurations as A, B, B, A so drift over time affects both equally.
- Clock throttling
A GPU lowering its SM clock for power or heat, which slows compute-bound work in proportion and bandwidth-bound work much less.
- Cache carry-over
A benchmark run hitting KV the previous run left in the prefix cache because the same prompts were resent.
- ignore_eos
A sampling flag that keeps generating past the end-of-sequence token, used by benchmarks to fix output lengths.
- Gauge versus counter
A gauge reports a current value and is sampled at scrape time; a counter only grows, and rates come from its difference over a window.
- Per-stream worst gap
The longest pause between tokens in one stream, reported as a distribution over streams.
Not the same as P99 ITL, which is taken over all gaps and cannot see a freeze that affects a minority of streams once each.
- Retokenized output tokens
A benchmark client's count of output tokens recomputed by tokenizing the text actually received.
Implement-from-Scratch Coding7 terms · questions
- Reference oracle
A slow, obviously correct implementation that a fast one is tested against on random inputs.
It must share none of the code under test: a dense masked attention, not the same gather, checks a sparse kernel.
- Property testing
Testing invariants over many generated inputs, such as conservation of blocks or a permutation that round-trips.
- Copy-on-write blocks
Sharing KV blocks by reference count and copying a block only when one sharer is about to write into it.
- Chained block hash
A block's cache key computed from its tokens and its parent block's hash, so equal keys imply equal prefixes.
- Token bucket
A rate limiter that refills at a fixed rate up to a burst capacity and admits work only while it has tokens.
- MoE routergate
The layer that picks each token's experts and their weights.
DeepSeek V3 scores with a sigmoid, adds a correction bias for selection only, keeps 4 of 8 expert groups, picks 8 experts and scales weights by 2.5.
- State snapshot
A saved copy of a recurrent layer's state at a position, the only way to resume or roll back a recurrence.
The same mechanism in vLLM and SGLang
Read from vLLM v0.32.0 and SGLang v0.5.22 source. Where the two names do not mean quite the same thing, the difference is in the last column; carrying a number across without it is how a tuned config stops working after a migration.
| Mechanism | vLLM | SGLang | Difference |
|---|---|---|---|
| Memory the engine may claim (gpu_memory_utilization) | gpu_memory_utilization | mem_fraction_static | vLLM's fraction covers everything the engine allocates; SGLang's covers weights and the KV pool. |
| Cap on running requests (max_num_seqs) | max_num_seqs | max_running_requests | Same idea. |
| Tokens per step (Token budget) | max_num_batched_tokens | chunked_prefill_size | vLLM's budget counts prefill and decode tokens together; SGLang's bounds the prompt chunk of a prefill pass. |
| Unit of KV allocation (Paged KV cache) | block_size (16 on CUDA) | page_size | SGLang resolves the page size per attention backend. |
| Prefix reuse (Radix cache) | automatic prefix caching | radix cache | vLLM matches whole 16-token blocks by chained hash; SGLang matches token spans in a tree. |
| Out of KV mid-decode (Preemption) | preemption (newest first) | retraction (fewest outputs first) | Same idea. |
| CPU work hidden behind the GPU (Overlap scheduling) | async_scheduling | overlap scheduler (--disable-overlap-schedule) | Same idea. |
| Release GPU memory without exiting (Sleep mode) | sleep mode (enable_sleep_mode) | release_memory_occupation | Same idea. |
| KV beyond the GPU pool (KV connector) | KV connector | HiCache (--enable-hierarchical-cache) | Same idea. |
| Data-parallel attention with expert-parallel MoE (DP attention) | data_parallel_size + enable_expert_parallel | --attn-dp-size + --ep-size | Same idea. |
For GPU execution terms (warps, register pressure, memory coalescing, profiler counters) with worked cases, the Inference Engineer’s Lexicon is the long-form reference. To see how these terms depend on one another, use the concept map.