The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.
an account raises the free count per topic
← Batching, Scheduling & Admission / 10
mediumNewRed HatAnyscaleBaseten
max_num_seqs or max_num_batched_tokens: which one binds, when, and how would you set max_num_seqs from first principles?
The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackDesign a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?Next in this trackvLLM turns on async scheduling by default. What does it overlap, what does it need from the model runner, and when must it be off?Next in this trackDesign a batch inference service for 10 billion tokens a day on a 70B model, optimizing for cost rather than latency.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.