xAI (SpaceXSI) Batching, Scheduling & Admission interview questions

Same arrival rate, same replica, same config: before the burst the queue held at 30, after it the queue grows by thousands. The answer finds the rule that makes one load have two steady states, proves it with a model, and picks the fix that removes the second state.

Same pool, same prompts, same config, and the number of running requests moves from 132 to about 189 in ten seconds, drops to 112 under the benchmark client and to 92 when clients stop sending max_tokens. The answer derives every one of those numbers from the scheduler's own rule.

One multiplication gives the number of sequences resident on the GPU, and from it the KV bytes and the batch the step runs at. The trap is that the duration in that multiplication depends on the answer.

Per token, moving KV over PCIe is several times cheaper than recomputing it, yet vLLM recomputes. The answer derives the break-even bandwidth, shows how small copies fall below it, and names what makes recompute nearly free.

A larger batch can raise tokens per second while fewer requests meet their latency targets. Work through the batch model, separate qualified request rate from capacity, and test how the SLO changes the answer.

The gain is not a constant. It is a statistic of your output-length distribution, and the right token budget is a point on the roofline. The answer derives both, then says where the paper number and the real knee disagree.

One 32K-token prompt in front of ten short ones costs each of them more than a second of TTFT. The answer separates the half of the job length you know from the half you do not, and schedules each differently.

A priority flag fixes the queue and leaves the real damage in place: batch sequences that are already running slow every interactive token. The answer puts a number on the KV the batch class may hold.

The 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.

Request-fair scheduling hands 96% of the GPU to the tenant with long documents. The answer derives the token weights from the roofline, shows the counter that enforces them per step, and the one rule that stops a tenant from banking credit.

Chunked prefill keeps the decoders running, so the damage looks bounded. The walkthrough shows the inter-token gap climbing from 10 ms to a full second, new arrivals locked out for 8.6 seconds, and a slowdown that outlives the prefill.

The default of 1,024 sequences is never reached by a 2K-context workload on an 8B model, and is past the compute ridge for a 256-token one. The answer finds the three ceilings and picks the lowest.

The two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.

Scheduling step n+1 before step n's tokens exist sounds impossible. The answer shows the placeholder trick that makes it work, what it costs at an end-of-sequence token, and the configurations where vLLM switches it off for you.

With no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.

Each draft token is a real token in the step's budget, and verification turns a bandwidth-bound step compute-bound sooner. The answer shows the accounting, the rollback that makes rejection cheap, and the batch size where speculation starts losing.

Accepting everything during a one-minute spike leaves the API serving dead requests for two minutes after it ends. The answer does the queue arithmetic, picks the rejection rule, and writes the response that keeps clients from making it worse.