Skip to content
← Batching, Scheduling & Admission / 09
mediumNewAnthropicGoogle DeepMindPerplexity

The scheduler admits a 100K-token prompt while 64 users are decoding. Walk through what happens to their inter-token latency, step by step.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackmax_num_seqs or max_num_batched_tokens: which one binds, when, and how would you set max_num_seqs from first principles?Next in this trackDesign a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?Next in this trackvLLM turns on async scheduling by default. What does it overlap, what does it need from the model runner, and when must it be off?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.