TL;DR: Meter quotas in replica time, not raw tokens: on Llama 3.3 70B (FP8, TP=4 H100, batch 128) an output token costs 110 us against 35.7 us for a prompt token, so weight outputs about 3.1x. Then isolate memory separately, because one 128K-token request holds 40 GiB of KV, 20% of a replica's pool. Fair queueing at the gateway, KV caps per tenant, and a per-tenant
cache_saltcover compute, memory and side channels.
How to approach it
Ask what tenants buy: a throughput contract, a latency SLO, or both, and whether any tenant needs hard isolation for compliance. Ask the spread of context lengths, since one long-context tenant changes the memory picture more than any rate. Then name what tenants share on one replica: step time, the KV pool and the prefix cache. Each needs its own control, and a rate limiter covers only the first.
A strong answer
Price the quota in the scarce resource. With assumed efficiencies of 50% MFU for prefill and 75% MBU for decode, at a decode batch of 128 and 1,700 tokens of context:
prompt token = 2 x 70.55e9 / (4 x 1979e12 x 0.50) = 35.7 us
decode step = (17.6 GB + 128 x 1,700 x 327,680 B / 4) / 2.51e12 = 14.1 ms
output token = 14.1 ms / 128 = 110 us
weight = 110 / 35.7 = 3.1 prompt-token units per output token
replica budget = 0.65 / 35.7 us = 18,232 units per second at a 65% ceiling
A tenant sending 5 req/s at 2,000 in and 500 out consumes:
units/s = 5 x (2,000 + 500 x 3.09) = 17,725, or 0.97 replicas
Raw-token limits would let a tenant shift to long outputs and triple its load without touching its quota. Prompt tokens served from the prefix cache cost almost no compute, so a discounted weight for them (the exact value is a pricing choice) rewards tenants for stable prefixes.
Compute isolation: fair queueing in front of a shallow engine queue. Each tenant gets a token bucket refilled at its contracted units per second, with a burst depth of, for example, 60 seconds of quota. Admitted requests enter per-tenant queues at the gateway, and a weighted fair scheduler drains them into each engine. The engine queue stays shallow (a couple of steps of work), because once a request is inside the engine its order is set by the engine policy: in vLLM v0.31.0 that is fcfs by default, with a priority policy (lower value first, arrival time breaking ties). Priority alone starves low tiers under sustained load, so fairness belongs at the gateway and priority only separates tiers.
Memory isolation: KV is a separate resource. The pool holds 646,503 tokens per replica, and a 128K-token context holds:
KV per token = 2 x 80 layers x 8 kv_heads x 128 x 2 B = 327,680 B
128K request = 131,072 x 327,680 B = 40.0 GiB, 10.7 GB per GPU at TP=4
share of pool = 131,072 / 646,503 = 20%
Five such requests fill a replica and force preemption of everyone else's running sequences, while that tenant's compute quota looks fine. So each tenant tier gets a cap on resident KV tokens per replica and a maximum context, and requests above, for example, 32K tokens route to a long-context pool sized for them.
Cache isolation. A shared prefix cache leaks: if tenant B's TTFT drops when it sends tenant A's system prompt, B learns A sent it. vLLM v0.31.0 mixes a per-request cache_salt into the hash of the first block (and LoRA identity into every block's hash), so a salt per tenant gives each tenant its own cache namespace. The cost is duplicate copies of prefixes that tenants legitimately share, which is why public, platform-owned prefixes can go unsalted.
Hard isolation is priced, not assumed. A dedicated replica is four GPUs, $7,200 a month at an illustrative $2.50 per GPU-hour, whether the tenant uses it or not. Offer it to tenants whose contract or regulator requires it, and run everyone else shared.
Oversubscription. Quotas sold can exceed fleet capacity because tenants do not peak together, but the ratio is an assumption to measure from the sum of per-tenant peaks against the fleet peak, never a constant chosen up front.
The reversal condition: with a handful of large tenants whose loads each exceed several replicas, sharing buys little statistical multiplexing and per-tenant replica pools are simpler and safer than fair queueing.
What interviewers probe next
"A tenant's retries create a thundering herd after a 429." Return a retry-after derived from the bucket's refill time and add jitter at the SDK; count rejected requests against nothing.
"How do you bill a request that is preempted and recomputed?" Bill the tokens the tenant asked for, not the work the platform redid. Recompute is the platform's cost and an alarm signal.
"A tenant needs guaranteed p99 TPOT." Quota is not enough; cap the decode batch on the replicas serving them, which is a capacity reservation priced like dedicated hardware.
How it's scored
Weak: proposes requests-per-minute limits per API key and assumes the engine shares fairly.
Hire: meters quotas in weighted tokens derived from prefill and decode cost, puts fair queueing at the gateway, and caps context per tier.
Strong Hire: treats KV memory and the prefix cache as separate shared resources with their own controls, explains the cache timing side channel and cache_salt, and prices dedicated isolation.
Common mistakes
- Charging prompt and output tokens equally in the quota, when an output token costs about 3x the replica time here.
- Letting a deep engine queue reorder work, so the gateway's fairness decisions are undone by FCFS inside the engine.
- Forgetting that long contexts consume memory, not compute, and preempt other tenants without exceeding any rate.
- Sharing the prefix cache across tenants with no salt, which exposes whether another customer sent a given prompt.