Twenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.
an account raises the free count per topic
← Inference System Design / 09
expertNewTogether AIFireworks AIBaseten
Design a platform serving 20 models of different sizes on one shared fleet. How do you place them, and when does a model get its own GPUs?
Twenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.
Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.
UP NEXT ON YOUR JOURNEY
Next in this trackPrice an LLM API: go from a GPU-hour cost to a per-million-token price for input and for output. Why do they differ?Next in this trackDesign the rollout of a new model version so that neither quality nor latency regresses in production. How do you know when to promote?Next in this trackDesign serving for a coding assistant: 32K-token prompts, 200-token completions, p99 TTFT under one second. What do you build?
DISCUSSION · 0
Nobody has commented on this answer. Add a correction or another way to work it.