Latency Decomposition: Queue, Prefill, Decode
How to break LLM request latency into its parts: TTFT is network plus queue plus prefill, end-to-end adds output length times inter-token latency. Which vLLM metrics measure each part, why percentiles do not add, and how Little's law cross-checks a dashboard.
TL;DR: Every latency complaint resolves into three numbers: time waiting to be scheduled, time in prefill, and time per decode step. TTFT is network plus queue plus prefill; end-to-end latency adds (output tokens minus 1) times inter-token latency, so 300 tokens at 30 ms each is 9 s regardless of how fast the first token came. Measure each part separately, because their fixes are unrelated and their percentiles do not add.
The identity
For one streamed request with output tokens:
Each term has a different owner. is the client, the gateway and anything before the server starts its clock. is admission: the scheduler had no room in the batch or the KV pool. is prompt compute, which is FLOP-bound and grows with prompt length minus cached prefix. The inter-token latencies (ITL) are decode steps, which are bandwidth-bound and grow with batch size, context length, and whatever prompt chunks share the step.
Which metric measures which term
vLLM v0.31 exports the parts as Prometheus histograms (checked October 2026):
| Term | vLLM metric | What moves it |
|---|---|---|
| TTFT, server side | vllm:time_to_first_token_seconds | everything below except the network |
| Queue | vllm:request_queue_time_seconds | load, max_num_seqs, KV pressure, preemptions |
| Prefill | vllm:request_prefill_time_seconds | uncached prompt tokens, token budget, MFU |
| Decode, whole | vllm:request_decode_time_seconds | output length times ITL |
| ITL | vllm:inter_token_latency_seconds | batch, context, prefill interference |
| Context | vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:num_preemptions | the state that explains the above |
Client TTFT minus server TTFT is the network and pre-engine time. If that gap grows, stop looking at GPUs.
Worked example
A TP=4 H100 SXM replica serving Llama 3.3 70B in FP8 at an assumed 50% prefill MFU gets 35.7 us of replica time per prompt token. Take a 2,000-token prompt, 300 output tokens, and a dashboard that shows 6 req/s arriving at this replica, a mean waiting queue of 3, and a measured mean ITL of 30 ms.
prefill floor = 2,000 x 35.7 us = 71 ms
queue (mean) = waiting / arrival = 3 / 6 req/s = 0.50 s (Little's law)
TTFT (mean) = 0.50 + 0.071 + network = about 0.58 s
decode = 299 x 30 ms = 8.97 s
e2e (mean) = 0.58 + 8.97 = about 9.6 s
running check = 6 req/s x (0.071 + 8.97 s) = 54 sequences in the batch
The last line is the cross-check: if vllm:num_requests_running reads 54, the histograms and the gauges agree. If it reads 120, requests stay longer than the histograms say (retries, abandoned streams the server keeps decoding, or a gauge scraped from a different replica). The decomposition says this request spends 6% of its life before the first token and 94% decoding; a TTFT SLO and an end-to-end SLO are fixed in different places.
Percentiles do not add
The p99 TTFT is not p99 queue plus p99 prefill. The slow queue and the slow prefill rarely land on the same request.
import numpy as np
rng = np.random.default_rng(0)
n = 200_000
queue = np.where(rng.random(n) < 0.8, 0.0, rng.exponential(0.25, n)) # s
prefill = rng.lognormal(np.log(0.08), 0.5, n) # s
p99 = lambda x: np.percentile(x, 99) * 1e3
print(f"p99 queue {p99(queue):.0f} ms + p99 prefill {p99(prefill):.0f} ms "
f"= {p99(queue) + p99(prefill):.0f} ms")
print(f"p99 of the sum (the real TTFT tail) {p99(queue + prefill):.0f} ms")
# p99 queue 749 ms + p99 prefill 257 ms = 1007 ms
# p99 of the sum (the real TTFT tail) 845 ms
Use per-request traces when you need the joint tail; use the histograms to see which component moved.
TPOT is not ITL
Two averages hide here. TPOT per request is , one number per request; ITL pools every gap from every request. vllm bench serve (v0.31.0) reports both: TPOT percentiles across requests, ITL percentiles across tokens. Long outputs contribute many gaps, so ITL weights them more heavily, and a stall of one step shows in p99 ITL while barely moving that request's TPOT.
Where it breaks
- Chunked prefill blurs the line. A prompt split over several steps shares those steps with decodes, so prefill time includes decode work and ITL includes prompt chunks. The token budget sets how much.
- Preemption counts twice. A preempted sequence goes back to waiting and, under recompute, prefills again; its queue and prefill time both inflate.
- Mean versus tail. A workload change shifts the whole distribution; a scheduling problem mostly moves the tail. Always read p50 and p99 together.
- Speculative decoding. ITL becomes bursty: several accepted tokens arrive at once, then a gap.
What interviewers listen for
That the candidate decomposes before hypothesizing, names the metric for each term, and knows which resource bounds each (queue: admission and KV, prefill: FLOPs, decode: bandwidth). Strong answers use Little's law to check one metric against another, refuse to add percentiles, and distinguish per-request TPOT from per-token ITL.
Say it in 30 seconds
"TTFT is network plus queue plus prefill, and end-to-end adds n minus 1 inter-token latencies, which usually dominate. I read each from its own histogram: queue time, prefill time, ITL. Queue points at admission or KV pressure, prefill at uncached prompt tokens, ITL at batch size and prefill interference. I sanity-check with Little's law, running sequences equal arrival rate times time in the batch, and I never add p99s."