Skip to content
← Batching, Scheduling & Admission / 08
hardNewTogether AIFireworks AIBaseten

Two tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackThe scheduler admits a 100K-token prompt while 64 users are decoding. Walk through what happens to their inter-token latency, step by step.Next in this trackmax_num_seqs or max_num_batched_tokens: which one binds, when, and how would you set max_num_seqs from first principles?Next in this trackDesign a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.