Skip to content
← Inference System Design / 06
hardNewOpenAIAnthropicGoogle

Design serving for a chat product and a batch API on the same GPU pool. How much batch work fits, and how do you protect chat latency?

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

A chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackDesign serving for 500 customer LoRA fine-tunes of one base model. Where do the adapters live, and how does a request find one?Next in this trackDesign a multi-region deployment for a 70B model that survives the loss of a whole region. How much spare capacity do you buy, and where?Next in this trackDesign a platform serving 20 models of different sizes on one shared fleet. How do you place them, and when does a model get its own GPUs?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.