12vLLM turns on async scheduling by default. What does it overlap, what does it need from the model runner, and when must it be off?▼hardNewRed HatNVIDIAAnyscale◆ premiumScheduling step n+1 before step n's tokens exist sounds impossible. The answer shows the placeholder trick that makes it work, what it costs at an end-of-sequence token, and the configurations where vLLM switches it off for you.Open full answer →
06What does SGLang's overlap scheduler hide, how does it build the next batch before the current tokens exist, and what does it cost?▼hardNewxAI (SpaceXSI)DatabricksTogether AI○ sign inThe GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.Open full answer →
13An 8B model on one H100 stops scaling at high request rates and the GPU is not saturated. Prove the CPU is the bottleneck, then fix it.▼hardNewFireworks AITogether AIBaseten◆ premiumSmall models make the GPU step so short that host work stops hiding behind it. The answer shows how to prove that with a timeline instead of a hunch, how much throughput a few milliseconds of Python cost, and which fixes help in which order.Open full answer →
16Structured output slowed generation. Is the cause grammar compilation, mask construction, or host scheduling?▼hardNewNVIDIAFireworks AITogether AI◆ premiumUse cold-schema and warm-batch traces to separate compilation from recurring stalls. Identify the exposed delay, choose a targeted experiment, and preserve output correctness.Open full answer →