Skip to content
← Batching, Scheduling & Admission / 07
hardNewOpenAIAnthropicPerplexity

Why does p99 TTFT shoot up once utilization passes about 70%? Model it as a queue, then say where that model is wrong for a batching engine.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackTwo tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?Next in this trackThe scheduler admits a 100K-token prompt while 64 users are decoding. Walk through what happens to their inter-token latency, step by step.Next in this trackmax_num_seqs or max_num_batched_tokens: which one binds, when, and how would you set max_num_seqs from first principles?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.