Skip to content
← Inference System Design / 19
hardNewModalBasetenTogether AI

New replicas take nine minutes to serve, so scaling is always late. Here is the startup log: where do the minutes go?

A nine-minute start turns every traffic ramp into a queue. The answer reads a startup log stage by stage, prices each stage against the floor its bytes or work allow, separates the stages a cache removes from the ones only a warm pool can, and says which fix pays first.

Updated Nov 2026 · Worked answers on inference mechanisms, with calculations, implementation details, and follow-up questions.

A nine-minute start turns every traffic ramp into a queue. The answer reads a startup log stage by stage, prices each stage against the floor its bytes or work allow, separates the stages a cache removes from the ones only a warm pool can, and says which fix pays first.

10 answers per topic instead of 5, plus saved progress and bookmarks · costs nothingor unlock all 213 remaining answers · $25

Every worked answer and derivation in all 12 topics, every concept page, the full practice-test bank, and every code drill’s edge-case tests and reference solution.

6 months · One payment · No auto-renewal

UP NEXT ON YOUR JOURNEY
Next in this trackPrice a prompt-cache hit from the server side: what it saves, what holding a prefix costs, why long retention costs moreNext in this trackShould small inference services share a GPU through MIG, MPS or time slicing? What isolation do they get?Popular right nowWhy is batch-1 decode on a 70B model bandwidth-bound, and what is its latency floor on two H100s?
DISCUSSION · 0

Nobody has commented on this answer. Add a correction or another way to work it.