vllm metrics

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

Prompts fit, but their outputs exhaust the pool. Work through a page-level memory ledger, choose the next diagnostic check, and test a concurrency fix against queueing and goodput.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

A dashboard of forty graphs is a dashboard nobody reads at 2 a.m. The answer builds eight panels in the order an incident unfolds, ties each to the action it decides, and shows how two of them check each other with Little's law.