scheduling

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Code wants eight draft tokens and chat wants two, and a fixed gamma serves both badly. The answer simulates three policies on a mixed workload, finds the gain is 5% with a cheap drafter and 19% with an expensive one, then explains why batching takes most of it away.

The GPU finishes a decode step and waits while Python appends tokens, checks stop strings and plans the next batch. The answer shows how the loop reorders that work, where the placeholder tokens live, and the request that pays for it.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Prompts fit, but their outputs exhaust the pool. Work through a page-level memory ledger, choose the next diagnostic check, and test a concurrency fix against queueing and goodput.