Skip to content
← Distributed & Disaggregated Inference / 25
hardNewxAI (SpaceXSI)Together AIFireworks AI

Two SGLang data-parallel ranks deliver well under twice one rank, and half the requests have a bad TTFT. Diagnose it.

Doubling an SGLang server to two data-parallel ranks raised throughput 41%, and the TTFT tail landed on half the requests. The answer finds the rank that started small, works out why round robin caps the pair at the weaker rank, and says why token-aware balancing only moves the queue.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Doubling an SGLang server to two data-parallel ranks raised throughput 41%, and the TTFT tail landed on half the requests. The answer finds the rank that started small, works out why round robin caps the pair at the weaker rank, and says why token-aware balancing only moves the queue.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackA video diffusion model runs out of memory on longer clips. What does sequence parallelism save, and what does it cost?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?Popular right nowDecode attention intensity is fixed at the query-to-KV head ratio. Derive that, then tell me what MLA does to it.
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.