ttft

Decode and prefill run the same weights through the same kernels, yet one waits on memory and the other on tensor cores. The answer finds the prompt length where they trade places and turns it into a TTFT.

Twenty lines of roofline arithmetic get within striking distance of a real server, and the interviewer knows it. What they are scoring is the list of places the model breaks, and how you would catch each one before a capacity plan rests on it.

The transfer looks like gigabytes per request until you divide by prefill time. Then the prompt length cancels, and the real question becomes which models and which GPUs break the link.

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

An 8k-token prompt on a 70B model leaves 2.7 GB of KV that has to cross the network before decode can start. The answer shows that the sustained bandwidth is modest, that the latency is the real problem, and which side of the link runs out first.

Tensor parallelism across nodes would all-reduce every activation over the network. Splitting the sequence instead sends only the KV, which for a GQA model is small. The answer sizes that traffic, finds when it hides behind attention, and fixes the causal-mask imbalance that naive splitting creates.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Voice moves the latency target from the first token to the first audible clause, and the largest item in the budget is not a model at all. The answer allocates 300 ms line by line, then sizes the LLM tier from it.

Inspect tokenized prompts and cache-counter deltas to diagnose a TTFT regression. A runnable prefix-boundary check and a routing follow-up test whether the evidence actually supports the diagnosis.

Reconstruct token arrival from a request-correlated trace. Separate transfer overlap from decode admission, and show how first-token delivery changes the metric that exposes a handoff stall.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.

An event can contain metadata, one token or several tokens. Audit a buffered stream without inventing token timestamps, and separate client launch delay from the engine's execution time.