Batching, Scheduling & Admission practice test
Static, dynamic and iteration-level batching, chunked prefill and the token budget, preemption by swap or recompute, priority and fairness, queueing arithmetic, and the goodput a scheduler is really optimising. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.
Sample items with answers
A static batch holds its slots until the longest member finishes, so a 50-token request idles for 450 steps while still occupying its slot. Iteration-level scheduling, from the Orca paper, makes scheduling decisions per step: completed sequences free their slots immediately and waiting requests are admitted into the next step. Padding is what static batching does, and sorting by prompt length helps prefill but leaves the output-length mismatch untouched. Separate streams would give up the weight sharing that makes batching worth doing.
TTFT runs from request arrival to the first streamed token, so it includes time in the queue, the prefill pass and the first sample. Under load, queueing is often the larger share, which is why TTFT is a scheduler metric as much as a kernel one. TPOT (also called inter-token latency when measured per gap) is the steady-state spacing of later tokens, set by the decode step time of the whole running batch. Dividing end-to-end latency by token count folds TTFT into TPOT and hides both.
After the test
Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: