A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.
an account raises the free count per topic
← Batching, Scheduling & Admission / 06
hardNewOpenAIAnthropicGoogle DeepMind
Interactive and batch traffic share one engine. How do you stop batch jobs from wrecking interactive TTFT and TPOT?
A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackWhy does p99 TTFT shoot up once utilization passes about 70%? Model it as a queue, then say where that model is wrong for a batching engine.Next in this trackTwo tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?Next in this trackThe scheduler admits a 100K-token prompt while 64 users are decoding. Walk through what happens to their inter-token latency, step by step.
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.