Inference engineering: from execution to evidence
Work through the decisions behind an inference service: model execution, measurement, memory, kernels, numerics, distributed serving, production and multimodal deadlines. Finish with a runnable state machine and a serving-system review.
A fast kernel can make a service slower if its workspace displaces the cache. A warm prefix can be the wrong routing destination when its worker is overloaded. A benchmark can report higher throughput while completing fewer requests within their deadlines. This course teaches the calculations and experiments that distinguish those cases.
What you will produce
Keep a serving notebook with five sections: the request contract, the resource ledger, correctness oracles, experiment records and the release decision. Each module adds evidence to that notebook. The final project asks you to defend a configuration and then revise it under a changed failure assumption.
The 24 lessons contain worked numerical cases, mechanism diagrams, changed-constraint exercises and retrieval checks. Six lessons are free. Python examples use only the standard library and run on a CPU. Their results establish arithmetic and state-machine behavior; device performance still requires a measured implementation on the intended hardware.
Choose how to study
| Your task | Start here | Work alongside it |
|---|---|---|
| Understand the execution path | Execution, then capacity | Concept map for individual mechanisms |
| Diagnose a worker | Runtime and numerics | Interactive experiments and coding drills |
| Design a service | Distributed and production | System-design questions |
| Serve non-text workloads | Modalities | Encoder, audio and diffusion cases linked in each lesson |
| Read or reproduce an engine result | Evidence | Reading inference engines |
Read sequentially for the full progression. Search arrivals can start at any lesson: the orientation names the required knowledge, and the annotated links supply the deeper derivation.
Depth without repeating the same page
Concepts remain the detailed explanations of individual mechanisms. This course combines them into decisions with competing constraints. Use Serving under pressure for a focused single-worker design sheet, or Inference for RL rollouts for policy provenance, replay and distributed rollout boundaries.
The scope is execution below the model API, including hardware feasibility and serving behavior. Training algorithms, application-level RAG, general cluster administration and cloud procurement are outside this course. Named implementations and primary research appear in dated source notes. All worked service numbers are synthetic unless a source explicitly supplies a measurement.
When you finish, you can
- Trace a request through model execution and measure qualified completions without hiding failures.
- Derive compute, memory and transfer budgets and turn them into admission and scheduling policies.
- Test attention, quantization and speculative decoding against explicit correctness contracts.
- Choose distributed partitions, cache placement and warm capacity from measured workload constraints.
- Apply separate deadlines and state budgets to encoders, streaming audio and diffusion.
- Implement a serving state-machine oracle and defend a release using an independent evidence dossier.
BEFORE YOU START · Comfort writing Python and reading tensor shapes. You should know what a matrix multiplication computes; the course derives the inference-specific budgets. No GPU is needed for the worked CPU exercises.
Syllabus
Specify and observe execution
Follow tensor shapes, define the request contract and preserve enough evidence to reproduce it.
Account for work and state
Use compute, memory and transfer ledgers to reject infeasible designs before benchmarking.
Make the runtime obey the contract
Choose a scheduling policy, preserve asynchronous ownership and qualify kernel optimizations.
Spend precision and speculative work
Compare quantization candidates, merge attention correctly and preserve a target distribution.
Move work across devices
Choose partitions, diagnose expert imbalance and price remote cache handoffs.
Operate a changing workload
Route with incomplete information, plan warm capacity and release isolated model revisions.
Serve more than text
Schedule encoder work, streaming audio and dependent denoising against their actual deadlines.
Build and defend a result
Design a reproduction, implement a state-machine oracle and defend an independent serving dossier.