Reading inference engines: prove what the code preserves
Follow a token through an inference engine, test the state that survives each step, and audit the evidence behind a performance claim. Six lessons turn source reading into a reproducible engineering review.
An inference repository becomes useful when you can predict what one invocation reads, changes and publishes. This course develops that skill through a sequence of small investigations. Start with the call path, establish a reference result, then introduce batch changes, speculative execution and measurement boundaries.
The six lessons take an estimated 102 minutes, including a 35-minute final review. The first two are free. Every lesson asks for an artifact: a state map, an equivalence matrix, a failure reproduction or an evidence-backed release decision. The examples use original synthetic inputs and executable CPU references. They are not production-engine defects or benchmark results.
Bring a repository and a notebook
The source reading route identifies specific starting points in Mini-SGLang, LLMs-from-scratch, Uzu and AIPerf, with supplementary material for GPU kernels and inference physics. You can complete the numerical exercises without installing those projects. Dated source pointers show where to compare the course's contracts with a real implementation.
Keep your findings in one review notebook. Record the repository commit, the path you followed, the invariant you expect and the smallest input that would violate it. The final lesson uses that notebook to decide whether a synthetic engine patch is ready to measure.
What completion demonstrates
You will have reviewed the boundaries between scheduler, model state and output measurement. You will not have implemented an entire production server or certified a GPU kernel. Those require device correctness checks, synchronization analysis and workload measurements beyond these CPU exercises.
When you finish, you can
- Map request identity, computed state and publication through an engine invocation.
- Design a cache-equivalence test that exposes wrong positions and masked prefixes.
- Verify batch compaction and speculative commit boundaries against independent references.
- Distinguish token generation, transport chunks and client timing before accepting a speedup.
BEFORE YOU START · Python, array shapes, basic attention and familiarity with prefill and decode. The exercises run as CPU models; GPU access is optional for the separate measurement extension.
Syllabus
Establish the reference
Trace one invocation and define what cached execution must preserve.
Preserve state across steps
Keep request identity and committed history correct as the work changes.
Audit the evidence
Separate client observations from kernel claims and make a release decision.