Serving engine chains: the pool, graphs, prefix hits, overlap and adapters

Engine rounds check whether you know what the defaults do to memory and time. Every default here names its version; check the release you run, since the engines ship every few weeks.

The pool and the max_model_len check

inputs   vLLM v0.32.0 on one H100 SXM 80GB, Llama 3.1 8B BF16
         CUDA total 85.02 GB x 0.92 = 78.22 GB budget
pool     (78.22 - 16.06 weights - 3.0 assumed activations) GB / 128 KiB
         = 451,355 tokens
check    startup requires one max_model_len request to fit:
         131,072 x 131,072 B = 16 GiB, well inside 59.2 GB
sanity   at this size the pool, not max_num_seqs, limits concurrency

max_model_len is a startup check against the pool as well as a request limit. Raising it past what the pool holds makes the engine refuse to start.

Drilled in: gpu_memory_utilization, max_model_len and max_num_seqs

CUDA graph padding

inputs   vLLM v0.32.0 on H100 captures 51 sizes up to 512 tokens
         a batch of 257 pads to the next captured size, 272
extra    15 rows x 2 x 8.03e9 FLOPs = 2.4e11 FLOPs = 0.24 ms at 989 TFLOP/s
bound    the step's weight read is 16.06 GB / 3.35 TB/s = 4.79 ms
sanity   in bandwidth-bound decode the padded rows ride on reads the step
         makes anyway

Padding is close to free in decode. Graph capture costs memory and startup time instead, and the memory comes out of the KV pool.

Drilled in: CUDA graph capture sizes and padding · CUDA graphs for decode

Prefix-cache hit tokens

inputs   vLLM v0.32.0, 16-token blocks, a 1,200-token prompt fully cached
blocks   1,200 / 16 = 75 full blocks
rule     the last prompt token must run to produce logits, so the block
         holding it is recomputed
hit      74 x 16 = 1,184 of 1,200 tokens
sanity   a "100% cached" prompt still runs one block of prefill

Each block's hash chains its parent's hash and extra keys such as a LoRA or image id, so one hash identifies the whole prefix up to that block.

Drilled in: vLLM prefix caching with chained hashes

Block against radix granularity

inputs   two prompts that diverge at token 300
vLLM     whole 16-token blocks: floor(300 / 16) x 16 = 288 tokens hit
SGLang   radix tree with page size 1: 300 tokens hit
sanity   12 tokens apart; the engines differ more in eviction and queue
         ordering than in match granularity

vLLM's content-addressed hashes are the same in every process, which is what lets a router or an external KV store use them.

Drilled in: SGLang radix cache against vLLM block hashing

The overlap scheduler

inputs   Llama 3.1 8B at batch 8; an assumed 1.5 ms of CPU work per step
serial   GPU work plus CPU work: 6.9 ms per step
overlap  batch k launches before batch k-1's results are processed,
         so the CPU work hides under the GPU: 5.4 ms per step
gain     6.9 / 5.4 = 1.28, about 28% more throughput
sanity   the gain is the CPU share of the serial step; small models at
         small batch gain most

The next batch's input tokens are not on the host yet when it launches. The GPU reads them from a buffer the previous forward wrote, indexed by request slot.

Drilled in: SGLang overlap scheduler

No-evict admission

inputs   TensorRT-LLM v1.2.1, default GUARANTEED_NO_EVICT
         a 70B FP8 replica on four H100s; 2,000-token prompts,
         max_tokens 2,000, real outputs of 300 tokens
rule     admit only if the pool holds prompt plus max_tokens
result   about 177 requests running, against about 329 for a policy
         that grows each request as it decodes
sanity   the reservation pays for 1,700 tokens per request that are
         never generated

The guarantee removes preemption and costs nearly half the concurrency on this mix. The v1.4.0 release candidates change the default, so set the policy explicitly.

Drilled in: TensorRT-LLM in 2026: when to pick it

Detokenization cost

inputs   a 2,000-token reply streamed token by token
re-decode  decoding the whole output at every token:
         1 + 2 + ... + 2,000 = 2,001,000 token decodes
window   decode a short window of recent tokens and emit only new text:
         a few tokens per step, about 2,000 windows in total
sanity   quadratic against linear, on the CPU that also schedules the batch

The window also handles characters split across tokens: while the decoded tail ends in U+FFFD, hold the text back until the next token completes it.

Drilled in: Incremental detokenization and multibyte UTF-8

Multi-LoRA bytes per step

inputs   Llama 3.1 8B, rank-16 adapters on all seven linear layers in each
         of 32 layers, BF16
per layer  rank x (in + out) summed: 16 x 81,920 = 1.31M parameters
adapter  32 x 1.31M x 2 B = 84 MB
8 adapters  in one decode step: 8 x 84 MB = 671 MB, 4.2% of the 16.06 GB
         base read
sanity   the FLOPs are 0.6% extra; the bytes per distinct adapter are what
         a decode step pays

vLLM v0.32.0 sets max_loras to 1 by default, which serializes requests for different adapters. Raise it to the number of adapters you expect in one batch.

Drilled in: Multi-LoRA batching with shrink and expand