Capacity Planning in Tokens per Second
How to size an LLM serving fleet: convert requests into prompt and output tokens per second, price prefill by FLOPs and decode by bytes, cap utilization for tail latency, check KV residency and TPOT, then turn replicas into a cost per million tokens.
TL;DR: A request rate is not a load. Convert it into prompt tokens per second (priced in FLOPs) and output tokens per second (priced in bytes), add the two as replica-seconds of work per second, divide by a utilization ceiling, then check that the KV cache fits and TPOT holds. For a 70B FP8 model on TP=2 H200 replicas at 50 req/s, 3,000 tokens in and 400 out, that is 16.7 replica-seconds per second, 26 replicas and 52 GPUs at 65% utilization.
Why requests per second is the wrong unit
Two requests can differ in cost by 7x on the replica below. A 30,000-token RAG prompt with a 50-token answer (2.15 replica-seconds) and a 200-token chat turn with a 1,000-token reply (0.31) hit different resources: the first spends its life in prefill, which saturates tensor cores, and the second in decode, which streams weights and KV from HBM once per token. Any plan that starts from "one GPU handles N requests per second" has hidden a workload assumption inside N, and it breaks the first time prompt lengths change.
The unit that survives is replica-seconds of work per second. Each phase gets its own cost per token, measured against the resource that bounds it, and the sum tells you how many replicas would be 100% busy. Then you add headroom.
The equation
For arrival rate , mean prompt length , mean output length , a model of parameters, replica peak , decode batch and step time :
The first term is prefill: FLOPs per prompt token at some fraction of peak. The second is decode: one step produces tokens and costs , where is bytes read (weights plus every running sequence's KV) over achieved bandwidth. is the utilization you are willing to run at, set by tail latency rather than by cost.
Worked example
Llama 3.3 70B (70.55B parameters computed from its config) with FP8 weights on TP=2 H200 SXM (1,979 TFLOP/s dense FP8, 4.8 TB/s, 141 GB each). Load: 50 req/s, 3,000 tokens in, 400 out. Assumptions, stated out loud: 50% MFU for prefill, 75% MBU for decode, BF16 KV, 0.92 memory fraction and 3 GB reserved per GPU.
load = 50 x 3,000 = 150,000 prompt tok/s ; 50 x 400 = 20,000 output tok/s
prefill/tok = 2 x 70.55e9 / (2 x 1,979e12 x 0.50) = 71.3 us of replica time
prefill = 150,000 x 71.3 us = 10.7 replica-s per s
TTFT floor = 3,000 x 71.3 us = 214 ms of compute
weights/GPU = 70.55 GB / 2 = 35.3 GB
KV/token = 2 x 80 x 8 x 128 x 2 B = 327,680 B ; per GPU at TP=2 = 163,840 B
KV/seq/GPU = 3,200 tokens (mean context mid-reply) x 163,840 B = 0.52 GB
KV pool/GPU = 141 x 0.92 - 3 - 35.3 = 91.4 GB -> at most 174 resident sequences
decode, B=64: bytes/GPU/step = 35.3 + 64 x 0.52 = 68.8 GB
t_step = 68.8e9 / (4.8e12 x 0.75) = 19.1 ms
replica = 64 / 19.1 ms = 3,347 tok/s
decode = 20,000 / 3,347 = 6.0 replica-s per s
total = 10.7 + 6.0 = 16.7 replica-s per s ; / 0.65 = 25.6 -> 26 replicas, 52 GPUs
cost = 52 x $3.50/h = $182/h (GPU-hour price illustrative)
= $182 / (20,000 x 3,600 / 1e6) = $2.53 per M output tokens
= $182 / (170,000 x 3,600 / 1e6) = $0.30 per M tokens, prompt included
Prefill is 64% of the work, so it is where the money goes, even though nobody watches prompt tokens stream.
The TPOT check picks the batch
Bigger decode batches buy throughput, but on a colocated replica the prompt chunks share the same GPUs. At 65% busy, decode steps are only part of the replica's time; in the worst case none of the prefill hides under decode memory time, and TPOT is the step time scaled by how much of the busy time is decode.
| Decode batch | Step floor at 75% MBU | GPUs at 65% | Worst-case TPOT | $ per M output |
|---|---|---|---|---|
| 32 | 14.5 ms | 62 | 32 ms | $3.01 |
| 64 | 19.1 ms | 52 | 53 ms | $2.53 |
| 128 | 28.4 ms | 48 | 97 ms | $2.33 |
| 192 | 37.8 ms | 46 | 140 ms | $2.24 |
With a 50 ms TPOT target, batch 32 is safe at any overlap, batch 64 needs a measurement showing some prefill hides, and batch 128 fails outright. Batch 192 would also overflow the 174-sequence KV pool. Going from 62 to 52 GPUs is a 16% saving that rides on one measured number.
Where it breaks
- MoE models. Prefill FLOPs use active parameters, but a large decode batch touches nearly every expert, so decode bytes use total parameters. Mixing those up misprices one phase by the sparsity ratio.
- Means hide tails. A p99 prompt 8x the mean costs 8x the prefill and owns the TTFT tail. Size a separate pool, or cap the per-request share of the token budget.
- Prefix caching. A hit removes prefill work. A 50% hit rate on prompt tokens halves the first term, which here is worth more than any decode tuning.
- Utilization at 90%. Queueing delay grows like ; a plan at 90% has no room for bursts or for the hour a replica is draining.
- MFU and MBU are inputs. Measure one replica at the planned chunk size and batch, then redo the sheet.
What interviewers listen for
They want the load in tokens per second for each phase before anything is drawn. They listen for prefill priced in FLOPs and decode in bytes, for the efficiency assumptions named and varied, for a KV residency check, and for TPOT including prefill interference. Strong candidates say which cost denominator they mean (output tokens or all tokens differ 8.5x here) and spot that prefix caching is the biggest lever when prompts dominate.
Say it in 30 seconds
"I convert requests into prompt and output tokens per second. Prefill costs 2N FLOPs per token at an assumed MFU; decode costs a step of weight plus KV bytes per batch of tokens at an assumed MBU. I add them as replica-seconds per second, divide by a 65% ceiling, then check the KV cache holds the batch and TPOT survives prompt chunks sharing the GPU. That gives replicas, GPUs and a cost per million tokens, and it shows prefill is most of the bill when prompts are long."