Use Little's law on an LLM server: how many requests are in flight at 10 per second, and does the KV cache hold them?

TL;DR: Little's law says requests in the system equal arrival rate times time in the system, $L = \lambda W$. At 10 req/s with 0.5 s TTFT and 300 output tokens at 25 ms each, $W = 7.98$ s and $L \approx 80$ requests, holding about 43 GB of KV on a 70B model. The catch is feedback: more concurrency slows every step, which raises $W$, so the naive "KV fits 33 req/s" estimate fills the cache at about 21.5 req/s instead.

How to approach it

State the law and its one requirement: a stable system, where the long-run arrival rate equals the completion rate. Then ask for what pins $W$ down: TTFT, output length and TPOT, measured at the operating point rather than from a single-user benchmark. Ask for the model and parallelism too, because the useful outputs of $L$ are KV bytes and batch size, and both need the model's per-token KV.

A strong answer

The law holds for any arrival process, any service order and any distribution of durations, as long as the averages are long-run averages. That generality is why it is the first tool for capacity questions: it needs no queueing model.

Work an example. Llama 3.3 70B on four H100s with tensor parallelism, prompts of 1,500 tokens, 300 output tokens, measured TTFT 0.5 s and TPOT 25 ms:

W            = TTFT + (output - 1) x TPOT = 0.5 + 299 x 0.025 = 7.975 s
L            = lambda x W = 10 x 7.975                       = 79.75 requests
decoding L_d = 10 x 299 x 0.025                               = 74.75 sequences
output rate  = L_d / TPOT = 74.75 / 0.025                     = 2,990 tok/s
cross-check  = lambda x 299 tokens                            = 2,990 tok/s

The cross-check matters. A decode step emits one token per running sequence, so concurrency divided by TPOT must equal the arrival rate times tokens per request. If a dashboard's numbers fail that identity, one of them is measured wrong, often TPOT averaged over tokens instead of requests.

Now size the KV the in-flight requests hold. Each request holds its prompt plus the tokens generated so far, on average 1,500 + 150 = 1,650 tokens:

KV per token (70B, GQA) = 2 x 80 layers x 8 kv_heads x 128 x 2 B = 320 KiB
KV in flight            = 79.75 x 1,650 x 327,680 B              = 43.1 GB
KV pool on TP=4         = 4 x (80 GB x 0.92 - 3 GB reserved) - 141.1 GB weights
                        = 141.3 GB = 431,200 tokens
naive max lambda        = 141.3 GB / (1,650 x 327,680 B x 7.975 s) = 33 req/s

The 33 is wrong, and saying why is the senior part of the answer. $W$ is not an input. Each decode step reads the weight shard plus every resident sequence's KV, so TPOT rises with $L_d$, which raises $W$, which raises $L$. Model it: the bandwidth floor on TP=4 is (35.3 GB + $L_d$ x 1,650 x 80 KiB) / 3.35 TB/s, 13.6 ms at 75 sequences, and the measured 25 ms is 1.85 times that floor. Hold that ratio and solve for the fixed point:

The pool fills near 21.5 req/s, at a TPOT near 38 ms. Past that, the scheduler preempts, recomputed prefill adds work, and $W$ grows faster still. Long before the pool fills, the TPOT objective probably fails: if the target is 30 ms, the same model puts the usable rate near 16 req/s, not 21.5 and not 33.

Three uses follow directly. Set max_num_seqs at or above the $L_d$ of the target operating point, or the engine caps concurrency before the KV pool does and the excess waits in the queue, inflating TTFT. Set router concurrency limits per replica from $L$ at the SLO. And autoscale on in-flight tokens rather than requests per second, since the KV identity is in tokens.

The reversal: when requests spend most of $W$ waiting in the queue rather than decoding, as during overload, $L$ is dominated by queued requests that hold no KV yet. Then Little's law still holds for the whole system, but KV must be computed from the running subset only, and the queue is a TTFT problem, not a memory problem.

What interviewers probe next

"Output length doubles to 600 tokens." $W$ roughly doubles, so $L$ and the KV in flight both roughly double at the same arrival rate, and the feedback through TPOT makes it somewhat worse than double.

"Does the law hold during a traffic spike?" Not instantaneously. It is a long-run identity; during a transient, arrivals exceed completions and $L$ grows without $W$ having been observed yet.

"Half the prompts share a 1,000-token system prompt with prefix caching." The shared blocks count once, so KV in flight falls by about 1,000 tokens for each of those requests after the first, while $L$ itself is unchanged.

How it's scored

Weak: multiplies arrival rate by output length and calls it concurrency, or uses single-request latency as $W$.

Hire: computes $L = \lambda W$ with TTFT and decode time, converts it to KV bytes with the per-token figure, and checks against the pool.

Strong Hire: cross-checks concurrency against tokens per second, recognizes that $W$ depends on $L$ through TPOT, and solves the fixed point instead of extrapolating linearly.

Common mistakes

• Using a TPOT measured at batch 1, which understates $W$ and therefore $L$ at production load.
• Charging each request its final length for its whole lifetime, which overstates KV by roughly the output length divided by two.
• Counting queued requests as KV holders. They hold none until admitted.
• Applying the law over a window shorter than the request duration, where arrivals and completions have not balanced.