Skip to content
← Inference System Design / 08
hardNewOpenAIAnthropicGoogle

Design a multi-region deployment for a 70B model that survives the loss of a whole region. How much spare capacity do you buy, and where?

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

Autoscaling cannot save a failover when a replica takes six minutes to come up, so the spare capacity has to exist before the region dies. The answer solves for the smallest fleet that survives any single region loss, and places the spare next to the region most likely to need it.

an account raises the free count per topic
UP NEXT ON YOUR JOURNEY
Next in this trackDesign a platform serving 20 models of different sizes on one shared fleet. How do you place them, and when does a model get its own GPUs?Next in this trackPrice an LLM API: go from a GPU-hour cost to a per-million-token price for input and for output. Why do they differ?Next in this trackDesign the rollout of a new model version so that neither quality nor latency regresses in production. How do you know when to promote?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.