debugging

A 2.5x gap at batch 1 almost never comes from the GPU being slow. The answer sorts the suspects by what a single profiler trace shows and puts a millisecond figure on each.

An error that grows with context length is a running sum losing increments, and BF16 stops adding 1 to a total at 256. The answer turns the symptom into a bisection, finds the state held in the wrong dtype, and shows the error curve that proves it.

Collectives run at the speed of their slowest link, so one NIC that trained at a quarter of its rate slows every GPU in the instance by the same amount. The answer shows the arithmetic that makes a 4x slower link a 2x slower step, then bisects the cluster to the culprit.

Compare three capture policies using a memory ledger and a matched workload. Recover the missing KV pages, then defend a policy against both capacity and latency constraints.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Users see the first token quickly and then the stream stutters. The answer builds the decode step from bytes, shows which cause produces a bimodal latency histogram, and names the one metric that tells the top two apart.

141 GB of weights on 160 GB of GPUs, and the engine refuses to start. The answer walks the five phases of engine startup, shows which one failed from the log, and does the arithmetic that says the model never fit the way it was configured.

FP8 halves the weight bytes, and the dashboard says the replica got slower. The answer computes what FP8 should have bought, splits the gap into kernel, scale and config terms with one profiler trace, and catches the regression that is not in the GPU at all.

Greedy decoding is deterministic on paper and the customer has two different answers in hand. The answer finds where the bits change, shows a measured example of it, and turns the finding into a promise the platform can keep.

Both numbers can be true at once. The answer shows how to turn the gap into a short list of measurable dilutions, which one usually eats most of the 2x, and the replay test that settles it.

A regular period is a clue most candidates throw away. The answer turns it into a ranked list of periodic sources, shows why a few slow steps out of hundreds own the p99, and names the one correlation that separates the top two.

A hang gives you no error, no log line and no metric. The answer shows how to make every rank say where it is stuck, why a wrong network transport makes things slow but rarely hangs them, and the order that finds the cause in minutes.

The speedup formula has two inputs that vary per request and per moment, and the tail lives where both are bad. The answer computes when speculation turns into a slowdown, shows which requests own the p99, and picks a policy instead of an on/off switch.

Small models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.

The KV pool is preallocated, so the growth is somewhere the engine does not budget. The answer splits memory into what is fixed and what can grow, turns a growth rate into bytes per request, and uses that one number to rule out half the suspects.

Use cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.

Compare expert counts with rank-level completion times. Explain why the longest combine span can occur on a lightly loaded rank, then test a placement change against a communication control.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Unused logical KV pages can coexist with exhausted device memory. Reconcile expanded image tokens, retained embeddings and phase-specific peaks before changing a decoder token limit.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

A runnable allocator hides an exception-safety defect and a payload-aliasing defect. Reduce each failure to a short operation sequence, then test ownership and content against separate oracles.