TL;DR: The load is 400,000 prompt tokens and 60,000 generated tokens per second. On TP=4 H100 replicas with FP8 weights (assuming 50% MFU for prefill and 75% MBU for decode), prefill costs 14.3 and decode 7.5 replica-seconds of work per second. Run at 65% utilization for tail latency, that is 34 replicas, 136 GPUs: roughly $1.57 per million output tokens at an illustrative $2.50 per GPU-hour. Prefill is 66% of the GPU time, so a 50% prefix-cache hit rate saves more GPUs (down to about 90) than any decode tuning.
How to approach it
Refuse to draw boxes until the load is in tokens per second for each phase, because prefill and decode are priced by different resources. Clarify: the distribution behind "2K in" (a long tail moves TTFT), how much of each prompt is shared (system prompt, tool schemas), whether output is streamed, and whether the 200 requests/s is a sustained peak or a burst. Then state every efficiency assumption out loud, since an interviewer cares more that they are named than about their exact values.
A strong answer
Load. prompt tokens/s and output tokens/s, a 6.7 to 1 ratio. Prefill is compute-bound, decode is bandwidth-bound, so each gets its own cost.
Replica choice. FP8 weights (70.6 GB) on TP=4 H100s: 17.6 GB of weights per GPU, leaving 53 GB for KV after the 0.92 memory fraction and a 3 GB reserve. At an average decoding context of 2,150 tokens a sequence holds 704 MB of KV, 176 MB per GPU, so a replica can keep about 300 sequences resident.
Prefill cost. Prefill needs FLOPs per token; at an assumed 50% of the FP8 peak across four GPUs:
per prompt token = 2 x 70.55e9 / (4 x 1979e12 x 0.50) = 35.7 us of replica time
fleet demand = 400,000 tok/s x 35.7 us = 14.3 replica-seconds per second
TTFT check = 2,000 tokens x 35.7 us = 71 ms of compute, plus queueing
Decode cost. A step reads the weights once plus every running sequence's KV, at an assumed 75% of peak bandwidth. At batch 128 per replica:
bytes per GPU per step = 17.6 GB + 128 x 176 MB = 40.2 GB
step (TPOT floor) = 40.2e9 / (3.35e12 x 0.75) = 16.0 ms
replica throughput = 128 / 16.0 ms = 8,000 tok/s
fleet demand = 60,000 / 8,000 = 7.5 replica-seconds per second
Fleet and cost. Total work is 21.8 replica-seconds per second. Queueing theory says latency tails grow steeply as utilization approaches 1, so cap it at 65%:
replicas = 21.8 / 0.65 = 33.5 -> 34 replicas = 136 GPUs (17 nodes of 8)
cost = 136 GPUs x $2.50/h = $340/h (price illustrative)
per M = $340 / (60,000 x 3,600 / 1e6) = $1.57 per million output tokens
$340 / (460,000 x 3,600 / 1e6) = $0.21 per million tokens, prompt included
| Batch per replica | TPOT floor | Whole TP=4 replicas | GPUs | $ per M output |
|---|---|---|---|---|
| 64 | 11.5 ms | 39 | 156 | $1.81 |
| 128 | 16.0 ms | 34 | 136 | $1.57 |
| 192 | 20.5 ms | 32 | 128 | $1.48 |
For each row, replicas are ceil((14.3 + 60,000 × step_seconds / batch) / 0.65). Multiply whole replicas by four GPUs, then divide hourly cost by 216 million output tokens/hour. These remain analytical estimates, not measured SLO capacities.
Larger decode batches save GPUs, but the decode step is not the whole TPOT. Prompt chunks share these GPUs through chunked prefill and take about 43% of each replica's wall time (14.3 of 33.5 replica-seconds), against 22% for decode steps. Mixed steps hide some prompt compute under each step's memory time, so TPOT lands between the 16 ms floor and a worst case where none is hidden:
batch 128: 16.0 ms x (65% busy / 22.4% decode) = 46 ms under 50 ms at worst
batch 192: 20.5 ms x (65% busy / 20.1% decode) = 66 ms fails unless chunks hide
So batch 128 is the defensible plan and 192 needs a measurement before anyone counts on it.
The architecture then follows the numbers: a gateway with per-tenant token budgets, a router that keeps requests sharing a prefix on the same replica, 34 colocated replicas with chunked prefill and prefix caching, and autoscaling on queued tokens and KV utilization rather than GPU utilization.
The lever that matters. Prefill is 66% of the GPU time. If half of each prompt is a shared system prompt and the router lands it on a replica that has it cached, prefill demand halves and the fleet falls to about 90 GPUs. Prefix-affinity routing is a cost decision before it is a latency one.
The reversal condition: disaggregating prefill and decode onto separate pools needs about the same GPU count here (88 for prefill, 48 for decode at the same 65% cap), so it is not a cost win at this ratio. It wins when the TPOT target tightens, because decode steps stop carrying prompt chunks, or when prompts get much longer than outputs.
If the interviewer gives one million daily users instead. Daily active users do not determine requests per second. For an original planning scenario, assume ten generations per user per day and a peak four times the daily mean, retaining this answer's 2,000/300-token mix:
requests/day = 1,000,000 x 10 = 10,000,000
average rate = 10,000,000 / 86,400 = 115.741 req/s
peak rate = 4 x 115.741 = 462.963 req/s
prompt/output = 925,926 input tok/s and 138,889 output tok/s at peak
work estimate = 21.8 x (462.963 / 200) = 50.463 replica-seconds/s
replicas = ceil(50.463 / 0.65) = 78; GPUs = 78 x 4 = 312
peak cost = 312 x $2.50 = $780/hour; $780 / 500 M output = $1.56/M
This carries over the assumed per-request service cost, not a guarantee that the larger deployment meets p99. Measure at the target load and add the chosen failure reserve. Use an hourly demand curve and billing granularity for daily spend; multiplying the peak by 24 assumes the entire day runs at peak.
What interviewers probe next
"Traffic doubles for an hour." Model loading takes minutes, so absorb bursts with the 35% headroom and admission control (shed or queue by tenant priority), and scale on a forecast, not on the spike.
"p99 prompt length is 16K." Those requests cost 8x the prefill and dominate TTFT tails. Cap the per-request share of the token budget and route long prompts to a pool sized for them.
"How do you check the 50% MFU assumption?" Measure prefill tokens per second at the planned chunk size on one replica before buying anything, and redo the sheet. The sheet is the method; the efficiencies are inputs.
How it's scored
Weak: sizes from requests per second with a per-GPU QPS guess, or forgets prefill entirely.
Hire: converts load to tokens per second per phase, prices prefill by FLOPs and decode by bytes, applies a utilization ceiling, and produces a GPU count and a cost per million tokens.
Strong Hire: states and varies the efficiency assumptions, checks KV residency and TPOT including prefill interference, identifies prefix caching as the largest lever, and explains when disaggregation does and does not pay.
Common mistakes
- Quoting cost per million output tokens and per million total tokens interchangeably. They differ 7.7x here.
- Running the fleet at 90% utilization in the plan, which turns p99 TTFT into a queueing problem.
- Treating prompt tokens as free because they are not streamed back.
- Autoscaling on GPU utilization, which reads high on a decode-bound fleet whether or not users are waiting.