Inference Performance Debugging Interview Questions

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

141 GB of weights on 160 GB of GPUs, and the engine refuses to start. The answer walks the five phases of engine startup, shows which one failed from the log, and does the arithmetic that says the model never fit the way it was configured.

Prompts fit, but their outputs exhaust the pool. Work through a page-level memory ledger, choose the next diagnostic check, and test a concurrency fix against queueing and goodput.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

Greedy decoding is deterministic on paper and the customer has two different answers in hand. The answer finds where the bits change, shows a measured example of it, and turns the finding into a promise the platform can keep.

Same model, same image, same instance type, and one replica drags the fleet's p99. The answer first proves the replica is slower at equal load rather than busier, then walks the hardware causes from GPU clocks to a link that trained at the wrong speed.

Perplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

A hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.

The speedup formula has two inputs that vary per request and per moment, and the tail lives where both are bad. The answer computes when speculation turns into a slowdown, shows which requests own the p99, and picks a policy instead of an on/off switch.

Small models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

A dashboard of forty graphs is a dashboard nobody reads at 2 a.m. The answer builds eight panels in the order an incident unfolds, ties each to the action it decides, and shows how two of them check each other with Little's law.

Use cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.

Compare expert counts with rank-level completion times. Explain why the longest combine span can occur on a lightly loaded rank, then test a placement change against a communication control.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

An event can contain metadata, one token or several tokens. Audit a buffered stream without inventing token timestamps, and separate client launch delay from the engine's execution time.