Debugging LLM serving, by symptom
44 symptoms an inference engineer is handed in production or in a debugging round. Each row gives the first measurement to take, the one that splits the likely causes, and links to the question that carries the diagnosis through. Taking the measurement before forming a theory is most of what a debugging round scores.
The first token is late
TTFT is queueing plus prefill plus one sampling step, so the first split is always whether requests waited or computed.
TTFT rose after a prompt-template deploy and GPU utilization did not move.
First check Tokenize an old and a new prompt and find where the token IDs diverge, then compare prefix-cache counter deltas across the deploy.
p99 TTFT shoots up once utilization passes about 70%.
First check Plot p99 TTFT against utilization alongside the variance of prompt lengths; the knee moves with that variance.
p99 TTFT Knee Above 70% Utilization free with sign-in
The prefix-cache hit rate is fine at night and collapses at peak.
First check Turn cache capacity into a survival time in seconds and measure how often each prefix comes back to the same replica.
After a short burst an SGLang replica on LPM never recovers, though traffic is back to normal.
First check Plot waiting-queue length and hit rate through the burst: at an unchanged arrival rate, a different steady state afterwards points at a rule, not at capacity.
Requests that hit the CPU KV tier reach the first token later than cold misses.
First check Time the hit path end to end rather than the copy; the transfer metric can look healthy while the hit is slow.
One DP-attention rank receives a long prompt and decode slows on every rank.
First check Compare per-rank step times inside one expert-parallel step instead of treating the ranks as independent replicas.
DP Attention Rank Imbalance Premium
The stream stutters after the first token
Inter-token problems hide in averages. Look at distributions per stream and per step before tuning anything.
The first token arrives quickly and then the stream stutters.
First check Plot the inter-token latency histogram and check whether it is bimodal.
p99 TPOT spikes on a regular period.
First check Measure the period exactly and list what in the system runs on that period.
Periodic p99 TPOT Spikes free with sign-in
Users report frozen streams, but P99 inter-token latency looks the same as the other engine's.
First check Report each stream's longest gap as a distribution over streams, next to the P99 taken over all gaps.
vLLM vs SGLang at a Full KV Pool Premium
Speculative decoding made p50 faster and p99 slower.
First check Break latency down per request by acceptance rate and by the batch size at the time, rather than as one fleet number.
Structured-output requests stall.
First check Trace a cold schema and a warm batch separately to tell a one-time delay from one that recurs.
The streamed chunks make token latency impossible to read.
First check Count tokens per streamed event before computing any per-token latency, and separate client delay from engine time.
The prefill-to-decode handoff in a disaggregated deployment adds latency.
First check Rebuild token arrival from a request-correlated trace across both pools.
Memory, preemption and startup
The KV pool is sized at startup, so most memory symptoms are a ledger that does not add up. Write the ledger before changing a flag.
The engine refuses to start although the weights fit in total GPU memory.
First check Find which startup phase failed in the log, then redo the per-GPU memory arithmetic for the configuration actually used.
Preemption is constant at peak while every prompt fits.
First check Build a page-level memory ledger that includes the output tokens each running request will still write.
More CUDA graph captures made decode faster and preemption more frequent.
First check Compare the KV pages each capture policy leaves, on a matched workload.
Memory grows slowly until the engine runs out.
First check Split memory into what is fixed at startup and what can grow, and turn the growth rate into bytes per request.
A multimodal model runs out of memory while KV pages are free.
First check Reconcile the expanded image-token count and the peak of each phase against device memory.
A multi-node deployment hangs on its first request with no error.
First check Make every rank report where it is blocked before changing any network setting.
Slower than the arithmetic says
Compute what the change should have bought first. A gap with a number on it can be split; a feeling cannot.
FP8 weights made the replica slower.
First check Compute what FP8 should have saved, then take one profiler trace and split the gap into kernel, scale and configuration terms.
An INT4 AWQ checkpoint runs slower than BF16 at batch 32.
First check Read the engine's kernel-selection log to see which GEMM kernel is running.
Same flags, new engine version, lower throughput.
First check Capture the resolved configuration on both versions and diff it.
A small model at a high request rate leaves the GPU idle.
First check Take a timeline and look for GPU idle time between steps while the host works.
Every kernel in the step is fast and the step is not.
First check Read the Nsight Systems timeline for launch gaps, synchronizations and CPU stalls, and price each one in milliseconds.
Reading a Decode Step in Nsight Systems free with sign-in
A decode GEMV reaches only half of HBM bandwidth.
First check Count the bytes each SM keeps in flight and compare with what the bandwidth needs.
A kernel runs slower on an H200 than on an H100.
First check Check whether the kernel's tuning table recognizes the new device, and reread the spec sheet for what did not grow.
Kernel Slower on H200 Than H100 Premium
All-reduce is a third of a TP=8 decode step.
First check Align the ranks in the trace and split each all-reduce into waiting and transfer.
An MoE step is slow and nobody knows whether it is hot experts or communication.
First check Compare token counts per expert with each rank's completion time.
One replica in eight drags the fleet's p99.
First check Prove it is slower at equal load, not busier, before looking at hardware.
Finding the One Slow Replica in Eight free with sign-in
The outputs changed
Rule out the inputs before the kernels: the same model given different token IDs or sampling settings is a different experiment.
Quality dropped and answers end early after an engine migration.
First check Compare the prompt token IDs and the resolved sampling settings between the two engines.
Temperature 0 gives a customer two different answers.
First check Find the first position where the two runs' logits differ, and what else was in the batch each time.
Nondeterminism at Temperature Zero free with sign-in
The same prompt gives different logits at batch 1 and batch 64 on an FP8 server.
First check Check whether any activation scale is computed over the whole batch rather than per request.
Batch Invariance and Logit Drift Premium
A quantized model matches baseline perplexity and fails long-context retrieval.
First check Run a small grid that isolates weight quantization, KV quantization and configuration, across context lengths.
Quantized Model Fails Long-Context Recall free with sign-in
Rollout log-probabilities disagree with the trainer's for an unchanged model.
First check Reconstruct the set of tokens each sampled probability was normalized over.
An attention kernel is exact in FP32 and wrong in BF16 at long context only.
First check Bisect over context length and look for state accumulated in a 16-bit type.
Speculative verification returns correct tokens and the next step diverges.
First check Trace what the cache commits on rejection, full acceptance, EOS and an output cap.
A seeded request changes output when the batch is reordered.
First check Check whether random draws are keyed by request and position or by launch order.
A double-buffered kernel fails a few CI runs and never under -G.
First check Run compute-sanitizer's race check on the release build.
Two numbers that cannot both be right
A benchmark measures a workload, a server state and a client. Before comparing results, check that all three were the same.
The benchmark says 2x and production shows 10%.
First check List the differences between the benchmark and production traffic that can each be measured, then replay production traffic.
Benchmark 2x, Production 10%: Reconcile free with sign-in
TTFT fell when the benchmark's concurrency doubled.
First check Find what state the previous run left on the server, and rerun on a freshly started one.
One run each, and the new config is 3.4% faster.
First check Run the same config twice to measure the noise, and record clocks for every run.
A/B Two Serving Configs Honestly Premium
Disaggregation won the benchmark and every request succeeded.
First check Cross-check the client's success and token counts against the server's own transfer counters.
Doubling rollout concurrency raised token throughput and cut usable samples.
First check Compare generated work with the samples the trainer can still consume.
Routing changes moved the hit rate and the tail in opposite directions.
First check Replay the same burst through each routing policy on the same trace.
Before trusting any of these measurements, check the benchmark that produced them: inference benchmark methodology covers open and closed loops, warm caches and coordinated omission. The terms used above are defined in the glossary.