expertNewOpenAIAnthropicGoogle DeepMind

Design a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY

Next in this trackvLLM turns on async scheduling by default. What does it overlap, what does it need from the model runner, and when must it be off?Next in this trackDesign a batch inference service for 10 billion tokens a day on a 70B model, optimizing for cost rather than latency.Next in this trackSpeculative decoding inside a continuous batch: how does the scheduler budget draft tokens, and what happens to the KV on rejection?

DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.