TL;DR: Size it before drawing it: 50 req/s at 1,000 tokens in and 250 out on Llama 3.1 8B (BF16, one H100 per replica) is 50,000 prompt and 12,500 output tokens per second, about 8 replicas at a 65% ceiling and roughly 250 open streams. Measure TTFT and inter-token latency at the gateway and at a client probe, never only inside the engine, because the engine sees about 32 ms of prefill compute while the user sees everything around it.
How to approach it
Ask for the load in tokens, not requests: arrival rate, prompt and output length distributions, and whether every response streams. Ask which latency the SLO is written against, the one a customer measures with a stopwatch or the one the engine logs. Then size the engine tier, because its replica count and the number of concurrent streams decide what the gateway and router must hold.
A strong answer
Load. The arithmetic runs on Llama 3.1 8B (8.03B parameters, 16.06 GB of BF16 weights) with assumed efficiencies of 50% MFU for prefill and 75% MBU for decode:
prompt tokens/s = 50 x 1,000 = 50,000
output tokens/s = 50 x 250 = 12,500
prefill per tok = 2 x 8.03e9 / (989e12 x 0.50) = 32.5 us
prefill demand = 50,000 x 32.5 us = 1.62 replica-seconds per second
TTFT compute = 1,000 x 32.5 us = 32 ms
Decode at a batch of 32 per replica, average context 1,125 tokens, with 128 KiB of KV per token (32 layers, 8 KV heads, head dim 128, BF16):
bytes per step = 16.06 GB + 32 x 1,125 x 131,072 B = 16.06 + 4.72 = 20.78 GB
step = 20.78e9 / (3.35e12 x 0.75) = 8.3 ms
replica tok/s = 32 / 8.3 ms = 3,869
decode demand = 12,500 / 3,869 = 3.23 replica-seconds per second
replicas = (1.62 + 3.23) / 0.65 = 7.5 -> 8 replicas, 8 H100s
cost = 8 x $2.50/h = $20/h (illustrative price)
per M output = $20 / (12,500 x 3,600 / 1e6) = $0.44
Streams in flight. Little's law gives concurrency: arrival rate times time in system. At a 20 ms TPOT a reply takes about 5 s, so:
open streams = 50 req/s x 5.0 s = 250, or 31 per replica (matches batch 32)
KV pool/replica = (80e9 x 0.92 - 3e9 - 16.06e9) / 131,072 B = 416,103 tokens
in use = 31 x 1,125 = 34,875 tokens, 8% of the pool
So this fleet is bound by decode bandwidth and the latency target, not by KV memory, and the gateway holds a few hundred long-lived connections, not thousands.
The path. Each hop owns one job and one clock:
- Gateway. Terminates TLS, authenticates, enforces per-tenant token budgets, counts prompt tokens for billing, and holds the stream. It records the first clock: request received.
- Router. Chooses a replica from live engine state (queued prompt tokens, KV usage, cached prefixes). Routing on request count alone fails here, since one 30K-token prompt costs as much prefill as thirty ordinary ones.
- Engine. Queues, prefills in chunks, decodes, streams deltas back. Its own metrics (in vLLM v0.31.0,
vllm:time_to_first_token_secondsandvllm:inter_token_latency_seconds) start the clock when the engine's API server process receives the request, after every hop in front of it. - Back out. The gateway relays each delta as a server-sent event and flushes immediately. A proxy with response buffering on will hold tokens and release them in bursts.
Where each SLO lives.
| Metric | Clock starts | Clock stops | Measured at |
|---|---|---|---|
| Engine TTFT | engine API server receives | first token sampled | engine histogram |
| Server TTFT | gateway receives | first SSE byte written | gateway |
| User TTFT | client sends | first byte arrives | synthetic probe |
| Inter-token latency | previous delta | next delta | gateway and probe |
Write the SLO against server TTFT at the gateway, alarm on the gap between it and engine TTFT (that gap is queueing, routing and auth), and use probes from each region to catch network and buffering. The engine's 32 ms of prefill is a small part of a user-visible TTFT that also carries queueing, TLS setup and a network round trip.
The reversal condition: if traffic is non-streaming batch work, TTFT stops mattering, the gateway holds no streams, and the SLO becomes end-to-end completion time at the gateway, which favors larger decode batches and fewer replicas.
What interviewers probe next
"Server TTFT p99 is 900 ms but engine TTFT p99 is 120 ms." The 780 ms is outside the engine: router queueing, a cold connection pool to the replica, or auth calls on the hot path. Trace the gateway first.
"Users say streaming stutters, ITL at the engine is flat 20 ms." Something between the engine and the client is buffering: proxy buffering, compression on the event stream, or a load balancer with a write coalescing timer.
"Prompt length p99 jumps to 32K." Prefill for that request is 32,000 x 32.5 us = 1.04 s of compute, so the router must account in tokens and cap one request's share of a step.
How it's scored
Weak: draws gateway, load balancer and GPUs with no load numbers, and quotes TTFT without saying where it was measured.
Hire: converts the load to tokens per phase, sizes replicas at a utilization ceiling, and places TTFT and ITL measurement at the gateway with a reason.
Strong Hire: uses Little's law to size concurrent streams and checks them against the KV pool, separates engine, server and user TTFT, and names buffering as the failure that hides behind a healthy engine.
Common mistakes
- Reporting the engine's TTFT histogram as the customer SLO, then missing a 700 ms router queue entirely.
- Load balancing on request count with a round-robin balancer, which treats a 32K prompt like a 200-token one.
- Leaving response buffering or gzip on for the event stream, so tokens arrive in clumps of dozens.
- Sizing gateway connections from requests per second rather than from rate times stream duration.