OpenAI Inference System Design interview questions

Most designs draw the router first. The answer that passes does the arithmetic first: tokens per second in each phase, replica-seconds per second, a utilization ceiling, then dollars, and finds that the prompt, not the reply, is two thirds of the bill.

Four boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.

Sending every request with the same prefix to the same replica halves the fleet, until one customer's prompt gets popular. The answer derives the spill threshold from the prefill a cache hit saves, in milliseconds.

Requests-per-minute limits let one tenant with 128K-token prompts take a fifth of a replica's memory per request. The answer prices quotas in the resource tokens actually consume, then isolates the three things tenants share: compute, KV memory and the prefix cache.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

Input and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.

A canary is only as good as the number of requests behind each verdict. The answer sizes the canary from the statistics of the regression it must catch, then stages the rollout around that number.

A cold 32K prefill on a 70B model barely fits a second of compute on eight H100s, before any queueing. The design that passes makes almost every request warm, then sizes two pools from what remains.

Voice moves the latency target from the first token to the first audible clause, and the largest item in the budget is not a model at all. The answer allocates 300 ms line by line, then sizes the LLM tier from it.

Counting GPUs across two generations gives the wrong plan, and routing by GPU count breaks the SLO on the slower pool. The answer defines one capacity unit at the SLO, then plans, routes and buys in it.

Offline work changes the regime: the batch is as big as memory allows, latency is irrelevant, and output tokens stop being expensive. The answer prices both passes in GPU-hours, finds which one is the bill, and cuts it.

A cheaper GPU and a faster decoder do not automatically produce a cheaper service. Audit a cost ledger, reject misleading wins, and choose a rollout only after quality and latency gates pass.

A web-service autoscaler watching GPU utilization will scale this fleet too late and in the wrong direction. The answer turns the four-minute load time into a headroom number, then decides what absorbs the bursts no scaler can catch.

Five hundred merged fine-tunes need five hundred GPUs before the first request. Served as adapters on a shared base they need a few dozen. The answer sizes the adapters, places them in a three-tier memory hierarchy, and finds the one setting that silently multiplies their memory by eight.

Twenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.