Question 1

Quantify what continuous batching buys over static batching, then pick the chunked-prefill token budget and defend it.

Accepted Answer

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

Question 2

What is goodput for an LLM server, and how can pushing throughput higher make it worse?

Accepted Answer

Doubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.

Question 3

Use Little's law on an LLM server: how many requests are in flight at 10 per second, and does the KV cache hold them?

Accepted Answer

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

Question 4

Preempt by recompute or by swapping KV to host memory? Derive the break-even from PCIe bandwidth and prefill speed.

Accepted Answer

Per token, moving KV over PCIe is several times cheaper than recomputing it, yet vLLM recomputes. The answer derives the break-even bandwidth, shows how small copies fall below it, and names what makes recompute nearly free.

Question 5

Where does head-of-line blocking come from under FCFS, when does shortest-job-first help, and how does it work without job lengths?

Accepted Answer

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

Question 6

Interactive and batch traffic share one engine. How do you stop batch jobs from wrecking interactive TTFT and TPOT?

Accepted Answer

A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.

Question 7

Why does p99 TTFT shoot up once utilization passes about 70%? Model it as a queue, then say where that model is wrong for a batching engine.

Accepted Answer

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Question 8

Two tenants share a replica. Should fairness count requests or tokens, and how does the scheduler enforce it step by step?

Accepted Answer

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

Question 9

The scheduler admits a 100K-token prompt while 64 users are decoding. Walk through what happens to their inter-token latency, step by step.

Accepted Answer

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

Question 10

max_num_seqs or max_num_batched_tokens: which one binds, when, and how would you set max_num_seqs from first principles?

Accepted Answer

The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.

Question 11

Design a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?

Accepted Answer

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Question 12

vLLM turns on async scheduling by default. What does it overlap, what does it need from the model runner, and when must it be off?

Accepted Answer

Scheduling step n+1 before step n's tokens exist sounds impossible. The answer shows the placeholder trick that makes it work, what it costs at an end-of-sequence token, and the configurations where vLLM switches it off for you.

Question 13

Design a batch inference service for 10 billion tokens a day on a 70B model, optimizing for cost rather than latency.

Accepted Answer

With no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.

Question 14

Speculative decoding inside a continuous batch: how does the scheduler budget draft tokens, and what happens to the KV on rejection?

Accepted Answer

Each draft token is a real token in the step's budget, and verification turns a bandwidth-bound step compute-bound sooner. The answer shows the accounting, the rollback that makes rejection cheap, and the batch size where speculation starts losing.

Question 15

Traffic triples for a minute on your LLM API. What do you reject, at what point, and what do you tell the client?

Accepted Answer

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.

Batching, Scheduling & Admission

Continuous Batching & Scheduling Interview Questions