How to read an inference engine: a route through code, kernels and measurement

Read an inference repository with a question you can answer from one execution. Which token is pending? Which request owns this row? What makes a cached position valid? A specific question gives you a stopping point and a test. A tour of every directory rarely does.

The route below connects the resources to engineering artifacts. Source paths were checked in October 2026. Pinned links preserve the versions inspected; a project's current branch may organize the same mechanism differently. The six-lesson course supplies original exercises, worked references and a final review using these ideas.

1. Establish what a cached forward pass preserves

Begin with Sebastian Raschka's LLMs-from-scratch KV-cache supplement. Follow the cached and uncached paths in gpt_with_kv_cache.py. Locate the mask's row offset, the position counters and both reset operations. Your artifact is a comparison matrix: empty cache, chunked append, single-token append and a second independent request after reset.

Use a small deterministic attention fixture before a model checkpoint. Distinct values and a poisoned unused tail expose the read boundary. Then compare corresponding full-prefix and cached logit rows with a stated tolerance. The course's cache-equivalence lesson shows how to construct that first fixture.

Train LLM From Scratch is a supplementary model-level reference. Read its modern attention and inference modules selectively. At this revision, src/inference/speculative.py explicitly recomputes the prefix to expose the sampling algorithm. That is useful for contrasting a reference algorithm with a serving implementation that must also preserve cached state. The pretraining and post-training pipeline is outside this route.

2. Follow one request through the scheduler

Mini-SGLang is the most direct next stop. Read python/minisgl/core.py, then _prepare_batch, _make_positions and _process_last_data in the scheduler. Record the distinction between request UID, table index, cached length and device length. Follow the saved batch into completion rather than assuming the current active list owns the result.

Your artifact is a launch record with request identity, input positions, KV locations and sampling settings. Reorder the next batch and explain why the old result still belongs to the old mapping. The batch compaction exercise provides an executable failure case with cancellation and external-ID reuse.

Pair this code with the Hugging Face continuous-batching documentation. Use the documentation to identify the public request lifecycle, then investigate how the implementation represents that lifecycle. Configuration examples are version-sensitive; record the installed package version before reproducing them.

For a larger architecture, read Aleksa Gordić's Inside vLLM. The article identifies its own August 2025 commit. Use its scheduler-to-forward path as a map, then verify any symbol or default against the revision you actually run. Your artifact should distinguish durable mechanisms from historical implementation details.

3. Inspect speculative state, not just acceptance

Uzu offers a different implementation context, including Metal-oriented execution and a DFlash speculator. Start with crates/uzu-engine/src/speculators/dflash_tfm.rs. Follow the accepted-index input and ask how it selects the state that survives verification. Do not assume a tree's accepted state can be represented by the same counter used by a chain.

Write separate ledgers for target state, draft state and emitted history. Then test rejection, a fully accepted draft, EOS inside an accepted prefix and the output-token cap. Our cache commit exercise supplies an original chain model for those boundaries.

The throughput numbers in a screenshot do not establish a portable speedup. A defensible comparison needs hardware, checkpoint format, workload, sampling settings, output counts, timing endpoints and repeated raw results. We have not reproduced Uzu's performance claims on Apple hardware. Its source is still useful for studying how state is represented and selected.

4. Connect a kernel to a falsifiable cost model

Read How To Scale Your Model: inference for the phase-level cost model. Keep hardware and precision assumptions attached to every calculation. The artifact is a predicted limiting resource for one specified shape, followed by a measurement that could contradict your prediction.

The Stanford CS336 Spring 2025 archive provides inference and systems lecture material in a broader language-modeling course. Use the inference lectures for the path from a forward pass to serving constraints. The MIT 6.5940 Fall 2024 course adds quantization and efficient-computing material. Select the relevant numerical and hardware sections; completing a training curriculum is not a prerequisite for this route.

Move from those models to the Triton language's fused-softmax tutorial. Inspect the input mask and the output mask separately. Predict what happens when an invalid padded lane contributes to the reduction. This is Triton the GPU programming language, distinct from NVIDIA's model-serving product.

GPU MODE's lecture archive complements that exercise. Lectures 1, 8 and 9 cover profiling, a CUDA performance checklist and reductions. Produce a correctness matrix before a timing chart. The site's RMSNorm investigation shows what an original experiment with retained raw evidence looks like, including a shape where the wider reduction loses.

5. Read the benchmark implementation

In AIPerf, start with arrival-pattern documentation and the metric implementation you intend to report. Follow the parser field that establishes first content and the source of token counts. Check how the first content chunk affects the per-token divisor. Your artifact is a short trace whose metrics you can calculate independently.

The streaming chunk exercise makes this concrete. It distinguishes intended arrival from actual send and separates content delivery from HTTP completion. A tool can report a valid aggregate while the client still experiences long gaps; the timestamps determine which claim is supported.

Use the LLM Inference Handbook as a vocabulary and planning reference alongside that trace. Its metrics, memory and optimization sections help identify the next question to investigate. Follow the implementation or original paper when the answer depends on a precise default, numerical guarantee or timing endpoint.

6. Audit the execution contract in DeepInfer

DeepInfer at commit 9bd9cf2 adds two useful assignments. In random_sampling.py, follow sequence seeds and sample positions into random draws. Then inspect the batch-invariance test: forcing a known token trajectory and comparing log-probabilities does not test free-running random-stream ownership. The sampling exercise separates those checks.

In csrc/engine/flops_estimator.hpp, follow query ranges into attention-pair accounting. The Context growth lab isolates the dense causal case: keep the chunk fixed and increase its cached prefix. Record when a linear-only budget predicts a pass but the context-dependent model misses.

The pinned release documents offline inference and additional components needed for a complete service. Its supported hardware and deterministic-mode conditions matter. We have inspected source, not reproduced its model execution or benchmark plots.

For further kernel work, Wafer's GPU performance resource map points to primary papers, specifications and implementations. Choose one question before following a branch. Its profiler references pair with Honest Kernel Benchmarking: explain how cache control and replay change the experiment before interpreting a reported speedup.

Broad curricula: select the serving material

AI Engineering from Scratch contains inference-optimization and metrics material within a much wider curriculum. Its searchable catalog supports goal-based navigation: choose an artifact to produce, then study the prerequisites for that artifact. Verify each runnable example against its declared model and independently recompute its numbers before adopting it.

Hands-On AI Engineering focuses largely on applications such as RAG, agents and OCR. Those projects can provide workloads, but their application architecture is outside this site's serving-path curriculum. For inference preparation, translate a workload into prompt lengths, output lengths, concurrency and correctness requirements, then return to the engine beneath it.

Ratishfolio's attention notes provide an introductory refresher. S30's public curriculum covers broader ML coding practice. Use prerequisite material when needed, then return to serving-specific tests. Neither a broad syllabus nor a large task count establishes depth in cache ownership, scheduling or kernel correctness.

Start the course when you have selected one source. Its final artifact is a review that identifies a failing invariant, demonstrates the failure and states the evidence required before accepting a performance change.