06Interactive and batch traffic share one engine. How do you stop batch jobs from wrecking interactive TTFT and TPOT?▼hardNewOpenAIAnthropicGoogle DeepMind○ sign inA priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.Open full answer →
08Two tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?▼hardNewTogether AIFireworks AIBaseten○ sign inRequest-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.Open full answer →