Compare vLLM, SGLang and TensorRT-LLM at pinned releases. Follow the KV budget, the request admission rule and the graph ceiling before choosing a configuration.
Start with Llama 3.1 8B on one NVIDIA H100 SXM: BF16 weights and KV, a 2,000-token prompt, a 2,000-token output limit and 300 tokens of mean output. Runtime memory is assumed to be 3 GB per GPU; memory already in use before SGLang loads is assumed to be 0.5 GiB. These are inputs you can change.
For this model, KV bytes per token = 2 × 32 layers × 8 KV heads × 128 head dimensions × 2 bytes = 131,072 bytes. The vLLM model leaves 59.16 GB per GPU for KV and estimates 209 concurrent requests at this mix. The live calculations below explain every subtraction and constraint.
This is an analytical capacity model. Decode timings are ideal lower bounds at dense peak or full memory bandwidth. Neither those timings nor the running counts are measured throughput or safe production concurrency.
Change the constraints
ILLUSTRATIVE MODEL · ALL EXPERIMENTS FREE
Engine memory plan
Same memory. Different promises.
vLLM v0.32.0
209running
451,330 KV tokens · memory binds
SGLang v0.5.22
121running
412,441 KV tokens · memory binds
TensorRT-LLM v1.2.1
113running
452,896 KV tokens · memory binds
Same GPU, model and traffic: vLLM v0.32.0 runs 209 requests and TensorRT-LLM v1.2.1 113. The pools differ far less than that; the admission rules make the gap.
At pinned defaults
vLLM v0.32.0
SGLang v0.5.22
TensorRT-LLM v1.2.1
KV pool per GPU
59.16 GB
54.06 GB
59.36 GB
KV tokens per replica
451,330
412,441
452,896
Page
16 tokens
1 token
32 tokens
Requests running at this mix
209
121
113
What binds
KV memory
KV memory
KV memory
Largest decode graph
512
256
128
Running batch in a graph
yes
yes
no: eager or piecewise
Step token budget
8,192
8,192
8,192
Decode step floor at that batch
22.38 ms
14.97 ms
14.30 ms
Aggregate output ceiling
9,341 tok/s
8,081 tok/s
7,902 tok/s
One 2,000-token prompt needs at least 32.5 ms of prefill compute on this replica at dense peak. Decode floors read weights (MoE: the experts the batch touches, uniform routing) plus each request's KV at a mean context of 2,150 tokens, at full memory bandwidth.
vLLM v0.32.0: the calculation and its sources
budget = ceil(79.18 GiB x 0.92) = 78.22 GB
pool = budget - 16.06 GB weights - 3 GB runtime (assumed) = 59.16 GB
tokens = pool / 131,072 B per token per GPU = 451,330
running = admits on the prompt and grows; at a mean context of 2,150 tokens, about 209 fit
Grow-as-you-go: a burst of long outputs preempts the newest request and recomputes it later. Decode graphs up to 512 sequences (51 sizes at 512); larger batches run piecewise or eager.
vllm/v1/worker/utils.py request_memory: budget = ceil(total x gpu_memory_utilization), default 0.92 (config/cache.py)
vllm/engine/arg_utils.py: max_num_seqs 1,024 at >= 70 GiB, else 256; API-server token budget 8,192 at 70 to 160 GiB, 16,384 at >= 160 GiB, 2,048 below
vllm/config/vllm.py _set_cudagraph_sizes: largest graph = min(2 x max_num_seqs, 512, or 1,024 on compute capability 10.x, token budget)
vllm/v1/kv_cache_interface.py SlidingWindowSpec: a sliding-window group holds about window / 16 + 1 blocks per request
SGLang v0.5.22: the calculation and its sources
capacity = 81,559 MiB (nvidia-smi)
reserve = 512 + 1.5 x 8192 + 1 / 8 x 1024 + 2 x 256 + 8 x 58 graphs, floor 10,240 = 13,904 MiB
fraction = (capacity - reserve) / capacity = 0.830
pool = free after weights - free before load x 0.170 = 54.06 GB, 412,441 tokens
running = 1 + (pool - 4,001 kept for the next admission) / (prompt + 0.7 x max_tokens) = 121; 176 once the forecast decays to 0.098
Forecast admission: about 121 after an idle period, 176 after 600 decode steps without a retraction. Decode graphs up to 256 requests; chunk 8,192; 58 prefill graph sizes.
arg_groups/memory_hook.py: chunk and decode max_bs by GPU memory; reserve = 512 + 1.5 x max(chunk, 2048) + TP x PP / 8 x 1024 + 2 x max_bs + 8 x prefill graph sizes MiB, floor 10,240 MiB above 60 GiB
model_executor/cuda_graph_config.py: breakable prefill graphs on by default on CUDA; environ.py SGLANG_ENABLE_POST_CAPTURE_KV_SIZING False
mem_cache/kv_cache_configurator.py _profile_available_bytes: pool = free after weights - free before load x (1 - mem_fraction_static)
managers/scheduler: a new request books prompt + min(max_new_tokens, 4,096) + 1; running ones book remaining output x new_token_ratio, 0.7 decaying to 0.098
mem_cache/kv_cache_configurator.py resolve_max_num_reqs: min(clamp(pool / context_len x 512, 2,048, 4,096), pool / 2)
TensorRT-LLM v1.2.1: the calculation and its sources
Reserve-to-completion: nothing is ever preempted, and the reservation for unused max_tokens is held for the whole request. Decode graphs for batch sizes 1 to 32, 64 and 128, without padding: any other batch runs eager.
llmapi/llm_args.py CudaGraphConfig: batch sizes 1 to 31, 32, 64, 128 by default, enable_padding False
A plan from each engine's defaults, not a benchmark. Running counts are steady-state estimates at the mean context; vLLM's grows until a burst preempts, SGLang's depends on how long its forecast has decayed, and TensorRT-LLM's never changes because it reserves max_tokens for every request. Prefix caching, speculative decoding (SGLang caps it at 48 running requests), quantization scales, LoRA and multimodal encoders are not modeled. Defaults move every release: check the tag you deploy.
NOW DEFEND THE RESULT
Three engines get almost the same KV pool on one H100. Why can one run nearly twice as many requests as another?
Decode and memory: dense Llama GQA with evenly sharded KV heads (TP 1, 2, 4 or 8). Linear matrix weights exclude input embedding lookups and norms. The latency model adds sequential linear and attention phase floors. It excludes launch gaps, communication, sampling, softmax overhead, activation traffic, KV writes and cache effects. FP8 is idealized native linear arithmetic; scale overhead, mixed-precision exceptions and quality loss are not modeled. Memory estimates include all model parameters at a uniform precision, BF16 KV, page rounding and a user-selected reservation.
Engine memory plan: vLLM v0.32.0, SGLang v0.5.22 and TensorRT-LLM v1.2.1 (PyTorch backend) at their default settings, read in each engine's tagged source; the derivation for each engine lists the files. Runtime memory and the CUDA context are user assumptions because the engines measure them. CUDA totals come from the hardware reference data below; SGLang's nvidia-smi capacity is sourced only for the H100 SXM. Hybrid sliding-window models are modeled for vLLM only.
Serving experiments: prefill deadlines use a specified linear cost curve. Prefix ownership counts full shared blocks and private tails. Goodput applies a request-level contract to an authored trace. All three are synthetic and do not reproduce a particular engine release. See the vLLM v0.32.0 scheduling discussion and prefix-cache design for implementation context.
Context growth: the exact pair count is for dense causal attention. Its time coefficients are synthetic, cover all layers and heads, and assume serial phases. DeepInfer’s FLOPs estimator, checked October 2026, is a source-reading assignment in context-sensitive accounting. It also handles compressed and windowed attention; this lab does not reproduce that model or its performance.
Hardware: dense vendor peaks, not achieved performance. Reference data checked 2026-11-03. The 90% budget and 4 GB reserve are illustrative inputs, not vLLM defaults. GB means 10⁹ bytes; KiB means 1,024 bytes.
Every example uses the baseline inputs above except the named change. Open its settings, inspect the arithmetic and share the calculator link. Results below are calculated from the same model as the interactive view.
One H100: the same model, three admission policies
A requested output ceiling of 2,000 tokens is much larger than the assumed mean output of 300. An engine that reserves the ceiling admits fewer requests than one that grows the allocation during decode. That difference is a memory promise; it does not establish which engine is faster.
Keep the mean output at 300 and reduce max_tokens to 300. TensorRT-LLM now reserves less unused headroom. Compare the running count with the baseline before attributing a concurrency difference to kernels.
Keep BF16 weights and change KV storage from BF16 to FP8. The model halves bytes per KV element. It does not include scale metadata, quantify accuracy loss, or certify a backend’s FP8 support; validate those separately before deployment.
The planner uses sourced CUDA-visible memory totals for the listed GPU parts. Startup activation peaks, graph allocations and other runtime memory vary with your configuration; replace the assumptions with measurements. It excludes prefix reuse, LoRA, multimodal encoders, speculative decoding and quantization scale overhead. Unsupported model/engine combinations are labeled “Not modeled.”
For an interview, explain which reservation changes and what observation could disprove your estimate. For deployment, measure request lengths, preemption and latency under representative load. A larger modeled batch is not a recommendation to send that much traffic.