TL;DR: Inter-token latency is the time of one engine step, so decompose the step: weight reads, KV reads, host overhead, and any prefill work the scheduler packed into it. For Llama 3.1 8B in BF16 at batch 64 and 4K context on an H100 SXM the bandwidth floor is 15.1 ms, and a step that also carries a 2,048-token prefill chunk takes about 79 ms. If the ITL histogram is bimodal with a p50 near the floor, prefill interference is the cause; if it shifts as a whole, the batch and context grew.
How to approach it
Fine TTFT and bad ITL says the queue and prefill are healthy and something is making individual decode steps slow or uneven. Before guessing, ask for the shape of the ITL distribution (p50 and p99, not the mean), the model, precision and GPU, the batch size and mean context, and whether chunked prefill is on. Chunked prefill is on by default in vLLM v0.31.0 (checked October 2026), so prefill and decode share steps unless someone turned it off.
A strong answer
One decode step reads every weight once and every live sequence's KV once, so start with the floor:
model: Llama 3.1 8B, BF16, one H100 SXM (3.35 TB/s HBM3)
weights = 8,030,261,248 params x 2 B = 16.06 GB -> 4.79 ms
KV reads = 64 seqs x 4,096 tokens x 131,072 B = 34.36 GB -> 10.26 ms
floor = (16.06 + 34.36) GB / 3.35 TB/s = 15.05 ms
sanity: a measured 24 ms p50 is 63% MBU, ordinary for a real server
Now add what the scheduler does. With chunked prefill, a new prompt is fed in slices that ride along with the running decodes. The step's matmuls then process 2,048 + 64 tokens instead of 64, which is far past the ridge point, so those GEMMs turn compute-bound:
ridge (BF16 dense) = 989 TFLOP/s / 3.35 TB/s = 295 FLOP/B
mixed step GEMMs = 2 x 8.03e9 x 2,112 tokens = 33.9 TFLOP
at 50% of 989 TFLOP/s dense = 68.6 ms
plus decode attention KV reads = 10.3 ms
step with a chunk = ~79 ms (5.2x the floor)
with a 512-token chunk: 2 x 8.03e9 x 576 / 494.5e12 + 10.3 ms = ~29 ms
Every running stream pays that 79 ms on every step that carries a chunk. That is the mechanism that keeps TTFT fine and makes ITL bad: the engine is buying fast first tokens with stalls for everyone else.
Rank the causes by how well they fit "TTFT fine, ITL bad":
| Rank | Cause | ITL histogram shape | Confirming signal |
|---|---|---|---|
| 1 | Prefill chunks in decode steps | bimodal: floor plus a 3x to 6x mode | spikes track new arrivals |
| 2 | Batch or context grew | whole distribution shifts right | p50 rises with vllm:num_requests_running |
| 3 | Host overhead (eager mode, slow sampling) | uniform offset, small batches hurt most | GPU idle gaps in an Nsight trace |
| 4 | Preemption | rare multi-second gaps | vllm:num_preemptions rising |
The metric that separates the top two is the vllm:inter_token_latency_seconds histogram, read as buckets rather than one quantile, set against the arrival rate. The rate of vllm:time_to_first_token_seconds observations counts requests that finished prefill, so it is a direct arrival signal. If the slow mode's share rises and falls with that rate while p50 sits near 15 ms, it is interference. If p50 itself climbs with vllm:num_requests_running and mean context, the floor moved: 128 sequences at 4K would put the KV term alone at 20.5 ms.
Fixes follow the cause. For interference, lower the per-step token budget (max_num_batched_tokens), which shrinks the slow mode from about 79 ms to about 29 ms at 512 tokens and costs TTFT, because a 4,096-token prompt now needs 8 chunked steps instead of 2. If both SLOs are tight, disaggregate prefill from decode so decode steps never carry a chunk. For a floor shift, cap max_num_seqs, quantize the KV cache to FP8 (halving the 10.3 ms term), or add replicas.
The reversal condition: if the histogram is unimodal and p50 is far above the bandwidth floor at small batch, stop looking at the scheduler. Take an Nsight Systems trace of a few steps; gaps between kernels mean the host is the bottleneck, which is a different fix (CUDA graphs on, sampling and detokenization off the critical path).
What interviewers probe next
"The product team says TTFT matters more than ITL. What do you change?" Keep a large chunk budget and accept the slow mode, but set an explicit ITL p99 SLO and alert on it, so the stalls are a measured trade rather than a surprise.
"Now it is a 70B model on TP=4." The same decomposition holds, with an extra all-reduce term per layer; the chunk's compute is split four ways, so the slow mode is smaller relative to a BF16 weight floor of 10.5 ms (35.3 GB per GPU at 3.35 TB/s).
"Why not just turn chunked prefill off?" Then a long prompt runs as one prefill step that stalls every decode for its full duration, which is worse for ITL p99, not better.
How it's scored
Weak: reports mean ITL, blames "GPU contention", and proposes more GPUs without computing a floor.
Hire: computes the weight and KV bandwidth floor, recognizes chunked prefill sharing steps with decode, and suggests tuning the token budget.
Strong Hire: reads the histogram shape, ties the slow mode to arrival rate with a specific metric, quantifies the chunk cost on both sides of the ridge point, and states the TTFT price of the fix.
Common mistakes
- Computing the decode floor from weights alone and forgetting that 64 sequences at 4K read twice the weight bytes in KV.
- Treating p99 ITL as a property of the slowest request, when it is a property of the steps that carried prefill work.
- Lowering the token budget without checking that TTFT p99 still meets its SLO.
- Reading GPU utilization, which sits near 100% in both modes and separates nothing.