Skip to content
← Batching, Scheduling & Admission / 19
hardNewxAIBasetenTogether AI

How many requests does SGLang admit into a fixed KV pool, why does that number climb under steady load, and when does it retract?

Same pool, same prompts, same config, and the number of running requests moves from 132 to about 189 in ten seconds, drops to 112 under the benchmark client and to 92 when clients stop sending max_tokens. The answer derives every one of those numbers from the scheduler's own rule.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Same pool, same prompts, same config, and the number of running requests moves from 132 to about 189 in ten seconds, drops to 112 under the benchmark client and to 92 when clients stop sending max_tokens. The answer derives every one of those numbers from the scheduler's own rule.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackAn embedding server handles short queries and long documents. How do you batch them without missing deadlines?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDecode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.