routing

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

Replay the same burst through affinity, round-robin and predicted-finish routing. Then remove local reservations or delay an eviction notice and explain exactly why the decisions change.