08Two tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?▼hardNewTogether AIFireworks AIBaseten○ sign inRequest-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.Open full answer →
05Design a multi-tenant LLM platform with per-tenant token quotas. How do you stop one tenant from degrading everyone else's latency?▼hardNewOpenAIAnthropicFireworks AIunlockedRequests-per-minute limits let one tenant with 128K-token prompts take a fifth of a replica's memory per request. The answer prices quotas in the resource tokens actually consume, then isolates the three things tenants share: compute, KV memory and the prefix cache.Open full answer →