TL;DR: Headroom equals ramp rate times lead time. With a 1 req/s-per-minute morning ramp and a 6-minute lead (detect, schedule, 4-minute load), the fleet needs 6 req/s of capacity in hand, under one replica of Llama 3.3 70B on four H100s. Scale on queued prompt tokens and KV usage against a forecast, keep the 65% utilization ceiling as burst absorber, and the fleet runs 568 instead of 888 replica-hours a day.
How to approach it
Ask for the traffic shape: the daily curve, how fast the steepest ramp climbs, and how large a burst arrives inside one minute. Ask where the four minutes go (image pull, weight download, GPU load, compile and graph capture), since part of it can usually be cut. Then compute one replica's capacity in requests per second, because every scaling threshold is a multiple of it.
A strong answer
Replica capacity. Llama 3.3 70B, FP8 weights on TP=4 H100, 1,500 tokens in and 400 out, with assumed 50% MFU for prefill and 75% MBU for decode at a batch of 128:
prefill per req = 1,500 x 2 x 70.55e9 / (4 x 1979e12 x 0.50) = 53.5 ms
decode step = (17.6 GB + 128 x 1,700 x 327,680 B / 4) / (3.35e12 x 0.75) = 14.1 ms
replica tok/s = 128 / 14.1 ms = 9,068
decode per req = 400 / 9,068 = 44.1 ms
work per req = 53.5 + 44.1 = 97.6 ms of replica time
capacity at 65% = 0.65 / 0.0976 s = 6.66 req/s per replica
KV check = 646,503-token pool vs 128 x 1,700 = 217,600 in use
Headroom from lead time. The fleet must already have, when a ramp starts, the capacity it will need when new replicas finish loading:
lead time = 1 min detect + 1 min schedule + 4 min load = 6 min
steepest ramp = 60 -> 240 req/s over 3 h = 1 req/s per minute
headroom = 1 req/s/min x 6 min = 6 req/s = 0.9 replicas
target replicas = ceil((load + 6) / 6.66)
So the scaler adds one replica of slack to whatever the current load needs and scales on where load will be in six minutes. A day with 8 hours each at 60, 150 and 240 req/s:
static at peak = ceil(246 / 6.66) = 37 replicas x 24 h = 888 replica-hours
autoscaled = (10 + 24 + 37) x 8 h = 568 replica-hours
GPU-hours = 3,552 vs 2,272, or $8,880 vs $5,680 per day at $2.50 (illustrative)
Bursts the scaler cannot catch. A 30% jump at peak inside one minute means 312 req/s, needing 47 replicas, six minutes away. The 37 replicas running can carry it only because the plan caps utilization at 65%:
37 replicas at 100% busy = 37 / 0.0976 s = 379 req/s
312 / 379 = 82% utilization: tails stretch, nothing drops
Beyond that, admission control takes over: shed or queue the lowest-priority tenants at the gateway rather than let every request's TTFT grow.
What to scale on. GPU utilization reads near 100% on any replica with a running decode batch, loaded or not, so it says nothing about demand. Scale on signals that measure waiting work: queued prompt tokens per replica (the router knows them; vLLM v0.31.0 exports vllm:num_requests_waiting and vllm:kv_cache_usage_perc per engine), KV usage above about 90%, and gateway TTFT p95. Scale up when forecast load plus headroom exceeds capacity, scale down slowly with a cooldown of 15 minutes or more, because a wrong scale-down costs six minutes of SLO breach to undo.
Cutting the lead time. The 4 minutes is worth attacking directly. Cache the container image and weights on node-local disk so scale-up skips downloads, and keep a warm pool of replicas in vLLM sleep mode (enable_sleep_mode, off by default in v0.31.0), whose level-1 sleep moves weights to host memory and frees the GPU for preemptible batch work. Waking copies 17.6 GB per GPU back, about 0.7 s at an assumed 25 GB/s over PCIe, instead of four minutes. Every minute removed from the lead shrinks the headroom by 1 req/s.
The reversal condition: if the lead time falls below the metric window (about 30 seconds) or traffic is flat, forecasting stops paying for its complexity and a reactive scaler on queued tokens is enough. If ramps are steeper than the 35% headroom can bridge for 6 minutes, buy static capacity instead.
What interviewers probe next
"A new replica passes its health check but the first requests are slow." Readiness must wait for weight load and CUDA graph capture to finish, and the router should ramp traffic onto it over tens of seconds, since a cold prefix cache sends every early request through full prefill.
"The cloud has no GPUs to give you." Capacity is a reservation problem, not an autoscaling one. Keep reserved nodes for the forecast peak and autoscale only replicas on them.
"Scale to zero at night?" Only for models whose users accept a 6-minute first request. For an interactive API, the floor is the minimum replicas that cover a regional failure.
How it's scored
Weak: sets a horizontal autoscaler on GPU utilization at 80% and calls it done.
Hire: computes per-replica capacity in requests per second, derives headroom from ramp rate times lead time, and scales on queue and KV signals.
Strong Hire: separates forecastable ramps from bursts, shows the utilization ceiling absorbing a 30% spike, adds admission control, and attacks the lead time itself.
Common mistakes
- Scaling on GPU utilization, which stays high on a decode-bound replica whether users are waiting or not.
- Marking a replica ready when the process starts, before weights load and graphs capture.
- Symmetric scale-up and scale-down thresholds, so the fleet oscillates around the boundary.
- Forgetting that new replicas arrive with empty prefix caches, so their effective capacity is lower for the first minutes.