Skip to content
← Batching, Scheduling & Admission / 17
hardNewxAIBasetenPerplexity

After a brief burst, an SGLang replica on LPM never recovers: traffic is back to normal and the queue still grows. Why?

Same arrival rate, same replica, same config: before the burst the queue held at 30, after it the queue grows by thousands. The answer finds the rule that makes one load have two steady states, proves it with a model, and picks the fix that removes the second state.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Same arrival rate, same replica, same config: before the burst the queue held at 30, after it the queue grows by thousands. The answer finds the rule that makes one load have two steady states, proves it with a model, and picks the fix that removes the second state.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackDoubling rollout concurrency raises token throughput but reduces usable samples. How do you diagnose the queue?Next in this trackHow many requests does SGLang admit into a fixed KV pool, why does that number climb under steady load, and when does it retract?Next in this trackAn embedding server handles short queries and long documents. How do you batch them without missing deadlines?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.