Performance debugging chains: what the symptom should have cost

A debugging round rewards the candidate who computes what the system should have done before guessing why it did not. Each chain below produces that expected number; the gap between it and the observation is the thing to explain. For routing a symptom to its first measurement, see debugging by symptom.

Lost prefix reuse

inputs   3,000-token prompts; before a template deploy, 95% of a 2,560-token
         shared prefix was cached: 243,200 cached tokens per 100 requests
before   cached per request = 2,432; computed = 3,000 - 2,432 = 568
after    the new template changes an early token: computed = 3,000
ratio    3,000 / 568 = 5.28x more prefill per request
sanity   the prompt length did not change; the reusable part did

A 5x rise in prefill work can raise queueing a great deal, but it does not predict the p99 ratio by itself. Measure the token IDs where old and new prompts diverge first.

Drilled in: TTFT jumped after a template deploy

The inter-token floor and a mixed step

inputs   Llama 3.1 8B BF16 on one H100 SXM, batch 64, 4,096 tokens of context
floor    (16.06 GB weights + 34.36 GB KV) / 3.35 TB/s = 15.05 ms
mixed    a 2,048-token prefill chunk in the step: 2 x 8.03e9 x 2,112 tokens
         = 33.9 TFLOP, 68.6 ms at 50% of peak, plus 10.3 ms of KV = ~79 ms
512 chunk  2 x 8.03e9 x 576 / 494.5e12 + 10.3 ms = ~29 ms
sanity   a measured 24 ms p50 is 63% of the floor's bandwidth, ordinary for
         a real server; a bimodal histogram is a different problem

Inter-token latency is the time of one engine step. Decompose the step and the cause of a bimodal histogram usually names itself.

Drilled in: Bad ITL with fine TTFT

A startup OOM that the totals hide

inputs   Llama 3.3 70B BF16 at TP=2 on H100 SXM 80GB, vLLM v0.32.0, 0.92
weights  70,553,706,496 x 2 B / 2 = 70.55 GB per GPU
budget   0.92 x 85.02 GB = 78.22 GB per GPU: 7.66 GB left for everything else
one seq  131,072 tokens x 163,840 B of KV per GPU = 21.47 GB
sanity   141 GB of weights on 160 GB of GPUs, and the engine cannot hold a
         single full-length sequence

"Fits" means weights, activation peak, runtime overhead and one max_model_len sequence inside the budget of each GPU. FP8 weights or TP=4 each leave about 41 GB per GPU for KV.

Drilled in: vLLM startup OOM walkthrough

A preemption page ledger

inputs   a synthetic 80e9 B device x 0.92 = 73.6e9 B; weights 16.06e9 B;
         an assumed 2.0e9 B for everything else; Llama 3.1 8B, 16-token pages
pool     (73.6 - 16.06 - 2.0)e9 / (16 x 131,072 B) = 26,483 pages
start    200 prompts of 2,048 tokens = 25,600 pages: they fit
finish   with 1,024 output tokens each = 200 x 192 = 38,400 pages
sanity   only 26,483 / 192 = 137 requests can reach full length together

The prompts fitting proves nothing about the outputs. Add each request's future output pages to the ledger before reading the preemption counter.

Drilled in: Constant preemption at peak

What FP8 should have bought

inputs   Llama 3.3 70B at TP=4 on H100 SXM, batch 64, 4,096 context, BF16 KV
BF16     35.28 GB weights + 21.47 GB KV per GPU = 16.94 ms per step
FP8      17.64 GB weights + 21.47 GB KV = 11.68 ms per step
expected 1.45x more throughput, not 2x: the KV did not shrink
observed a 15% loss puts the FP8 step near 16.94 / 0.85 = 19.93 ms,
         1.71x what it should be
sanity   scale overhead is about 1 ms; a 1.7x gap is a kernel or a config

Splitting the gap needs one fixed-batch profiler trace per configuration. The expected step time is what makes the trace readable.

Drilled in: FP8 throughput regression

A link that trained at the wrong speed

inputs   Llama 3.3 70B at TP=4, BF16 activations, batch 64, 4,096 context
bytes    2 x 80 layers x 2 x (4 - 1) / 4 x 8,192 x 2 B = 3.93 MB per token per GPU
step     64 x 3.93 MB = 251.7 MB of all-reduce per GPU per step
NVLink   at about 360 GB/s achieved: 0.70 ms; step 16.94 + 0.70 = 17.64 ms
PCIe     at about 50 GB/s achieved: 5.03 ms; step 21.97 ms
sanity   1.25x slower, and in a TP group the slowest GPU sets the pace for
         all four

Before suspecting hardware, prove the replica is slower at equal load: compare inter-token latency per replica at matched running-request counts.

Drilled in: One slow replica in eight

Benchmark 2x, production 10%

inputs   a change that halves weight bytes on Llama 3.3 70B at TP=4
batch 1  640-token context: 10.55 ms -> 5.28 ms per step, 2.00x
batch 64 4,000-token context: 16.79 ms -> 11.53 ms, 1.46x
Amdahl   prefill 40%, decode 60% of production GPU time, measured step gain 1.2x:
         1 / (0.40 + 0.60 / 1.2) = 1.11x end to end
sanity   both numbers can be right; they measure different workloads

Write production speedup as Amdahl over the phases the change touches, using the phase shares from production histograms, then replay production traffic to confirm.

Drilled in: Benchmark 2x, production 10%

A periodic p99 with a flat median

inputs   Llama 3.3 70B BF16 at TP=4, batch 32, 2,000 tokens of mean context
plain    (35.28 + 5.24) GB / 3.35 TB/s = 12.1 ms per step
mixed    a 2,048-token chunk: 2 x 70.55e9 x 2,048 / 4 / 989e12 = 73 ms at peak,
         so the step is at least 85 ms, 7.0x
period   one 16,384-token prompt every 5 s = 8 chunks; (5 - 8 x 0.085) / 12.1 ms
         = 357 normal steps
sanity   8 / 365 = 2.2% of steps are slow: the p99 lands on them, the p50
         never does

A regular period is evidence. List what in the system runs on that period, then correlate slow steps with it.

Drilled in: Periodic p99 TPOT spikes