Serving under pressure: from admission to release
Build a defensible serving policy for one specified worker: reserve KV pages, protect token deadlines, reclaim cancelled work safely, and decide whether a change deserves release.
A worker can have free device memory and still admit a request it cannot finish. A scheduler can improve prompt completion while making every active stream miss its next-token deadline. This course follows those failures through one small, specified serving system.
Six lessons connect decisions that are usually taught separately. The first two are free. The remaining lessons develop the execution policy, evidence review and final incident. Exercises use synthetic inputs with worked calculations; they are not measurements of a named engine or GPU.
What you will build
Keep a single design sheet as you work. Add the admission ledger, the chunk rule, the cancellation state machine and the release criteria. The last lesson gives you conflicting symptoms and asks you to change that design with evidence. Its rubric grades the decisions you can demonstrate.
Use the prefill deadline experiment, prefix ownership experiment and goodput experiment alongside the lessons. They expose the assumptions so you can change one constraint at a time.
The scope
The worker serves a dense decoder with a paged KV pool. There is one execution worker, one scheduler and no remote KV transfer. Memory is accounted in binary units. The service-cost model is deliberately synthetic. Production engines add interactions that this course asks you to measure before applying its policy.
When you finish, you can
- Write an admission rule that includes future output and shared-page ownership.
- Choose a prefill chunk from a token-gap budget and identify when the model fails.
- Separate cancellation from safe reclamation and prevent cross-scope prefix reuse.
- Grade a serving change using a fixed workload, request-level SLOs and a rollback rule.
BEFORE YOU START · Comfort with arrays, asynchronous execution and basic transformer inference. Read the free KV budget concept if cache sizing is new.
Syllabus
Make admission a promise
Specify the request contract and allocate against future occupancy.
Keep execution and ownership consistent
Trade prompt progress against token gaps, then release only what is safe.
Defend the release
Qualify completed requests and diagnose a serving regression from evidence.