Skip to content
Practice tests · 30 items

Inference System Design practice test

Design an LLM API on a fixed GPU pool: prefix and KV-aware routing, autoscaling on the right signal, multi-LoRA and multi-model serving, tenant isolation, cold starts and weight distribution, and cost per million tokens under an SLO. The pool for this topic holds 10 easy, 12 medium and 8 hard items. Each answer comes with its reasoning and a link to the question or concept page it was drawn from.

Configure a test
Topic
Difficulty
Questions
10 in this pool · about 7 min
Reveal answers
Sign in to startNeeds a free account · retakes deal unseen items first

Sample items with answers

easy · sample
Why should an LLM API measure time to first token at the gateway or with a client probe, not only inside the engine?
Engine timers run on the GPU clock, which drifts from wall-clock time under heavy decode load
✓The engine never sees network time, gateway queueing or tokenization, yet users wait for them
The engine reports TTFT only for requests that hit the prefix cache, so misses are never counted
Gateways sample every request while engines log one request in a hundred to save CPU time

TTFT as a user feels it is network plus gateway and router queueing plus engine queueing plus prefill plus the first token's trip back. The engine's own timer starts when the request reaches its scheduler, so it misses everything upstream. For an 8B model on one H100, a 1,000-token prompt is about 32 ms of prefill compute, and a slow gateway or a cold TLS handshake can exceed that. Measure at both points and alert on the gap, because it shows where the time went.

easy · sample
Why is GPU utilization a poor signal for autoscaling an LLM serving fleet?
✓It reads near 100% whenever a batch runs, and it cannot see requests waiting in queue
It updates only once a minute, which is too slow to react to bursts of incoming traffic
It counts tensor-core FLOPs only, so bandwidth-bound decode always shows close to 0%
It averages across GPUs in a node, which hides a single overloaded tensor-parallel rank

The utilization counter reports the fraction of time some kernel is running, not how much of the GPU that kernel uses. A replica decoding a batch of 4 and one decoding a batch of 200 both read near 100%, and neither number says how many users are waiting. Scale on demanded work instead: prompt and output token rates converted to replica-seconds, with queue depth and KV usage as guardrails and p99 TTFT as the check. Utilization still helps spot a dead or idle replica.

After the test

Picking an option is easier than deriving the answer at a whiteboard. These pages make you derive it: