13Design a batch inference service for 10 billion tokens a day on a 70B model, optimizing for cost rather than latency.▼mediumNewAnyscaleDatabricksSnowflake◆ premiumWith no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.Open full answer →
06Design serving for a chat product and a batch API on the same GPU pool. How much batch work fits, and how do you protect chat latency?▼hardNewOpenAIAnthropicGoogle○ sign inA chat fleet sized for its peak sits more than half idle over a day, and a batch API can soak it up. The answer measures the idle replica-hours, finds what limits batch throughput (it is not compute), and decides where the two workloads meet.Open full answer →
15Design batch inference over 1 billion documents: embed each one and run a 70B model over each. How many GPUs, how long, what cost?▼hardNewOpenAIDatabricksSnowflake◆ premiumOffline work changes the regime: the batch is as big as memory allows, latency is irrelevant, and output tokens stop being expensive. The answer prices both passes in GPU-hours, finds which one is the bill, and cuts it.Open full answer →