Skip to content
← Batching, Scheduling & Admission / 13
mediumNewAnyscaleDatabricksSnowflake

Design a batch inference service for 10 billion tokens a day on a 70B model, optimizing for cost rather than latency.

With no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.

Updated Oct 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

With no latency target, every scheduler knob points the other way from chat serving. The answer sizes the fleet from the roofline, finds that prefill is three quarters of the bill, and lists the levers in order of dollars saved.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 136 remaining answers · ₹2,000 / $25

Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackSpeculative decoding inside a continuous batch: how does the scheduler budget draft tokens, and what happens to the KV on rejection?Next in this trackTraffic triples for a minute on your LLM API. What do you reject, at what point, and what do you tell the client?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.