Inference for RL rollouts: preserve the training signal
Build an auditable contract between an inference engine and its trainer. Six lessons connect sampled-token records, teacher scoring, queue freshness and weight updates through interactive experiments and a final worker review.
A rollout engine supplies training data while its own model keeps changing. A generated answer is useful only if the trainer can identify the decisions inside it, reconstruct their probabilities and decide whether they are still eligible. This course follows that contract from the first token to a weight update.
The six lessons take an estimated 105 minutes, including a 30-minute final review. The first two are free; the remaining four use Premium access. Keep one review notebook as you work: a workload map, a token record, a memory ledger, a queue policy and a cache-transition checklist. The final lesson combines them in a synthetic incident with an observable acceptance rubric.
Experiments you can inspect
Change temperature and token filtering in the probability inspector. Make an unchanged model appear to have updated by storing the wrong probability, then repair the record. In the queue simulator, step through version publication and compare consumed samples with discarded work. Both experiments disclose their simplifying assumptions and use original inputs.
The course develops serving mechanisms rather than a recipe for training an agent. It does not tune a reward function, run distributed optimization or claim a GPU speedup. Pinned implementation pointers connect the exercises to real code without making the lessons depend on a changing default.
Continue from the existing curriculum
The public rollout-serving article maps the supplied Hugging Face curriculum to inference workloads. Serving under pressure develops admission and request-lifetime reasoning; Reading inference engines develops source-level correctness reviews. This course applies those skills to a trainer consuming outputs from a changing policy.
When you finish, you can
- Separate sequential generation from known-token scoring and account for their different outputs.
- Reconstruct a sampled token's probability from its history, policy version and sampling support.
- Budget teacher output buffers and bound stale work before spending inference capacity.
- Review a weight transition and reject a rollout worker with inconsistent records or cache state.
BEFORE YOU START · Python, causal attention, prefill and decode. The course introduces the post-training terms it uses. All exercises are synthetic CPU models; no training cluster or model download is required.
Syllabus
Identify the work and its evidence
Map generation and scoring, then preserve the token record the trainer needs.
Spend capacity on usable work
Budget vocabulary outputs and limit samples that will age out before consumption.
Review a changing policy
Define the cache boundary for a weight update, then audit a complete rollout worker.