Inference system design chains: demand, replicas, headroom and routing

A design round starts with arithmetic that converts traffic into replicas before any box is drawn. These chains follow one method: price each token in replica time by the resource it uses, add up the demand, then divide by what a replica can give under its latency target.

Prefill demand in replica-seconds

inputs   200 requests/s, 2,000 prompt and 300 output tokens each
         Llama 3.3 70B FP8 on TP=4 H100 SXM, 50% MFU assumed for prefill
demand   200 x 2,000 = 400,000 prompt tokens/s; 200 x 300 = 60,000 output tokens/s
per token  2 x 70.55e9 / (4 x 1,979e12 x 0.50) = 35.7 us of replica time
fleet    400,000 x 35.7 us = 14.3 replica-seconds per second
sanity   one 2,000-token prompt is 71 ms of compute before any queueing

Replica-seconds per second is a replica count at 100% busy. Every later step only adds the headroom that latency targets need.

Drilled in: Size the fleet: 70B chat at 200 requests/s · Capacity planning in tokens

Decode work and the TPOT share

inputs   the same replica: 17.6 GB of FP8 weights per GPU, 75% MBU assumed
weights  17.6 GB / (3.35e12 x 0.75) = 7.02 ms per step, whatever the batch
KV       176 MB per sequence per GPU / (3.35e12 x 0.75) = 70 us per sequence
fleet KV 60,000 output tokens/s x 70 us = 4.2 replica-seconds per second
share    at a 50 ms TPOT target, 1 - 7.02 / 50 = 0.86 of each replica is
         left for work that grows with traffic
sanity   the weight read is a fixed tax on every step, paid whatever the load

The weight read is paid once per step and shared by the whole batch. The KV read grows with every running sequence, which is why it counts as demand.

Drilled in: Size the fleet: 70B chat at 200 requests/s

Replicas at a utilization ceiling

inputs   demand 14.3 + 4.2 = 18.5 replica-seconds per second (18.47 unrounded)
         share 0.86 from the TPOT target; a 65% utilization ceiling for bursts
replicas 18.47 / (0.86 x 0.65) = 33.05 -> 34 replicas = 136 GPUs
check    60,000 / 34 = 1,765 output tokens/s per replica, about 27 sequences
sanity   a 1.58x burst takes TPOT to the 50 ms target before anything drops

Round up whole replicas before pricing anything. A fraction of a TP=4 replica cannot be bought.

Drilled in: Size the fleet: 70B chat at 200 requests/s

The prefix-affinity spill rule

inputs   Llama 3.3 70B FP8 on TP=2 H200 SXM, 50% MFU; 8,000-token prompts
         with 6,000 tokens shared per tenant prefix
per token  2 x 70.55e9 / (2 x 1,979e12 x 0.50) = 71.3 us
hit      6,000 x 71.3 us = 428 ms of prefill saved on the warm replica
rule     route to the warm replica unless its queue is more than 428 ms of
         work deeper than the coldest one
sanity   the rule compares prefill work, so affinity and load balance are
         measured in the same unit

Scoring replicas by the prefill work a request would add, queued prompt tokens plus this request's uncached tokens, makes the trade explicit instead of a tuned weight.

Drilled in: Prefix-affinity router with a load spill rule · Prefix-aware routing

Headroom for a cold start

inputs   lead time 1 min to detect + 1 min to schedule + 4 min to load = 6 min
         steepest ramp 1 request/s per minute; 7.39 requests/s per replica
headroom ramp x lead time = 1 x 6 = 6 requests/s = 0.81 replicas
target   ceil((load + 6) / 7.39) replicas
sanity   scaling on a forecast with this headroom ran 520 replica-hours a
         day against 816 for a fleet sized statically at peak

Headroom is set by how fast load rises and how long a replica takes to arrive, not by a percentage chosen in advance. Shorten the load and the headroom shrinks with it.

Drilled in: Autoscale with a four-minute cold start · Autoscaling LLM fleets

Quota units for prompt and output tokens

inputs   Llama 3.3 70B FP8 on TP=4 H100 SXM
prompt   35.7 us of compute per token, as above
output   the KV it reads each step: context x 327,680 B / 4 / 2.51e12 B/s
         = context x 32.6 ns; at 1,700 tokens of context, 55.4 us
weight   55.4 / 35.7 = 1.55 prompt-token units per output token at 1,700;
         context / 1,093 in general, 30 at 32K
sanity   a flat per-token price undercharges long-context output by an
         order of magnitude

Meter tenants in replica time. Counting raw tokens lets a long-context tenant consume many times its share at the same token count.

Drilled in: Multi-tenant quotas and isolation

One long request's share of a pool

inputs   Llama 3.3 70B, 320 KiB of KV per token in BF16; TP=4 H100 pool of
         702,879 tokens
request  131,072 x 327,680 B = 40.0 GiB, 10.7 GB per GPU
share    131,072 / 702,879 = 19% of the replica's pool
sanity   five such requests fill the pool; a per-tenant KV cap is a memory
         limit that rate limits do not provide

Isolation needs a memory limit as well as a rate limit. A tenant within its token quota can still hold a fifth of a replica's cache with one request.

Drilled in: Multi-tenant quotas and isolation

Price per phase from a GPU-hour

inputs   Llama 3.1 8B BF16 on one H100 SXM at an illustrative $2.50 per hour
prefill  2 x 8.03e9 / (989e12 x 0.50) = 32.5 us per token: 30,790 tokens/s
         $2.50 / (30,790 x 3,600) x 1e6 = $0.023 per million input tokens
decode   batch 128 at 1,500 tokens of context: 41.2 GB per step at 75% MBU
         is 16.4 ms, 7,801 tokens/s: $0.089 per million output tokens
ratio    0.089 / 0.023 = 3.9x
sanity   each phase is priced by the resource it saturates: FLOPs for input,
         bytes for output

These are costs at full utilization. Divide by the fleet's average utilization and add the margin before comparing with a list price.

Drilled in: Price an API from a GPU-hour