Every token.
Every tradeoff.
Go into your inference engineering interview with the intuition to derive, debug, and defend your decisions.
Start reading. No account required.
A course to follow.
A system to understand.
4 courses, 43 lessons. Start with the full inference engineering course, or choose a focused course for the work in front of you.
Browse the curriculumInference engineering: from execution to evidence
Work through the decisions behind an inference service: model execution, measurement, memory, kernels, numerics, distributed serving, production and multimodal deadlines. Finish with a runnable state machine and a serving-system review.
- 01Specify and observe execution
- 02Account for work and state
- 03Make the runtime obey the contract
- 04Spend precision and speculative work
- 05Move work across devices
- 06Operate a changing workload
- 07Serve more than text
- 08Build and defend a result
Free lessons need no account. Sign in to save reading progress; each lesson shows its access level.
Serving under pressure: from admission to release
Build a defensible serving policy for one specified worker: reserve KV pages, protect token deadlines, reclaim cancelled work safely, and decide whether a change deserves release.
6 LESSONS · 2 FREEReading inference engines: prove what the code preserves
Follow a token through an inference engine, test the state that survives each step, and audit the evidence behind a performance claim. Six lessons turn source reading into a reproducible engineering review.
6 LESSONS · 2 FREEInference for RL rollouts: preserve the training signal
Build an auditable contract between an inference engine and its trainer. Six lessons connect sampled-token records, teacher scoring, queue freshness and weight updates through interactive experiments and a final worker review.
Learn it. Question it.
Make it your own.
A derivation gives you the model. An experiment exposes its limits. A hard question asks you to make the call.
Get beneath the abstraction.
Memory layouts, kernel mechanics, and the decisions a serving engine makes for you.
Explore the libraryChange an input. Follow the cost.
Explore decode, KV memory, speculation, and scheduling with interactive teaching models.
Open an experimentTurn intuition into an answer.
Work the problem, compare your reasoning, and tackle the follow-up that changes a constraint.
Find your next questionYour role.
Your next level of depth.
Start with what your work demands. Follow a deliberate sequence through concepts, experiments, and worked questions.
Explore the learning pathsInference & serving
Build the memory model, follow the scheduler, and reason through an overloaded engine.
11 steps · Access shown at each step
- Bring with you
- Transformer basics and working Python.
- Work toward
- Diagnose capacity regressions and verify KV ownership, cancellation, and cache isolation.
- A good fit if
- You can explain attention and want to reason about what an engine does with memory and time once requests arrive.
- Not in this path
- Writing kernels and splitting a model across GPUs; the kernel path and the distributed-inference topic cover those.
- 01Concept · Free
The Roofline Model for LLM Inference
Separate arithmetic and memory limits.
- 02Concept · Free
The KV Cache: Bytes per Token and the Memory Budget
Account for the bytes a sequence retains.
- 03Experiment · Free
KV memory budget
Change context and concurrency in the memory model.
- 04Concept · Free
Iteration-Level Scheduling (Continuous Batching)
Trace which requests run in each iteration.
- View all 11 steps
Write it. Run it. Explain it.
Implement a serving component in Python, run it in your browser and inspect the result. Start with the split-KV merge; compare plans when you want the full edge-case suites and reference solutions.
The prompt is short.
The reasoning goes deep.
The weights fit.
The prompts fit.
So why the preemption?
At peak load, the engine keeps evicting sequences. Work from a concrete memory budget and decide what you would measure or change first.
Investigate the case01 Weights stay resident.
02 Prompt KV fills the pool.
03 Decode keeps adding state.
What did admission forget?
Conceptual memory sketch. Open the case for the synthetic evidence and arithmetic.
One serving path.
Twelve ways to go deeper.
Build the cost model.
Defend the decisions.
Work it through.
Keep it close.
You should be able
to check the reasoning.
Show the arithmetic. Name the hardware and the regime. Keep the source attached. When an assumption changes, the answer should change with it.
Read our accuracy standard



