02What is goodput for an LLM server, and how can pushing throughput higher make it worse?▼easyNewAnthropicOpenAIFireworks AIunlockedDoubling the batch raised tokens per second by a third and sent goodput to zero. The answer shows why every sequence in a batch shares one step time, and how to find the batch size that maximizes the number users feel.Open full answer →
11Design a scheduler that admits a request only if it predicts the TTFT and TPOT targets will both hold. What does it predict, and how?▼expertNewOpenAIAnthropicGoogle DeepMind◆ premiumThe two targets are coupled: the TPOT target caps the prefill chunk, which caps prefill throughput, which caps the backlog the TTFT target can absorb. The answer builds the step model, derives the admission rule and shows what to do with requests that fail it.Open full answer →
02Design the request path for an LLM API: gateway, router, engines, streaming. Where is each SLO measured, and what does each hop cost?▼easyNewOpenAIAnthropicTogether AIunlockedFour boxes are easy to draw. The answer that passes sizes them first, then says exactly which clock starts where, because a TTFT measured inside the engine and a TTFT measured by the client can disagree by a factor of five.Open full answer →
04Design autoscaling for an LLM fleet whose model takes four minutes to load. What do you scale on, and how much headroom do you hold?▼mediumNewBasetenModalTogether AIunlockedA web-service autoscaler watching GPU utilization will scale this fleet too late and in the wrong direction. The answer turns the four-minute load time into a headroom number, then decides what absorbs the bursts no scaler can catch.Open full answer →