07Why does p99 TTFT shoot up once utilization passes about 70%? Model it as a queue, then say where that model is wrong for a batching engine.▼hardNewOpenAIAnthropicPerplexity○ sign inThe 70% rule of thumb falls out of one formula once you plug in a prefill time and a p99 target. The answer derives it, shows how prompt-length variance moves it to 50%, and lists the four ways an LLM engine breaks the textbook queue.Open full answer →
09Design a platform serving 20 models of different sizes on one shared fleet. How do you place them, and when does a model get its own GPUs?▼expertNewTogether AIFireworks AIBaseten○ sign inTwenty models with two replicas each is a fleet that is mostly idle. The answer measures how idle, then tiers the models by load so the long tail shares GPUs without hurting the models that pay the bills.Open full answer →
10Price an LLM API: go from a GPU-hour cost to a per-million-token price for input and for output. Why do they differ?▼mediumNewOpenAIAnthropicTogether AI○ sign inInput and output tokens burn different resources, so they get different prices. The answer prices each phase from first principles, then shows which assumption moves the list price most.Open full answer →