RL rollout inference: what the serving engine must preserve
Learn how RL rollout serving differs from ordinary text generation: sampled-token probabilities, teacher scoring, stale queues and weight updates. Explore interactive models and a six-lesson inference course.
BY VIKAS MISHRA · INFERENCEINTERVIEWS EDITORIAL · UPDATED OCTOBER 29, 2026 · 5 MIN READ
PRACTICE THIS:Start the rollout inference course ·Understand behavior-policy provenance ·Practice serving-engine interviews
An RL rollout engine must return evidence about how it generated its tokens. The trainer needs the token history, the relevant policy revision and probabilities with a declared sampling meaning. Long conversations introduce tool waits; asynchronous training introduces sample age; weight updates introduce a cache boundary. A high generated-token rate is useful only when the resulting records remain valid and usable.
Our new Inference for RL rollouts course follows those contracts through six lessons. It starts with a workload map and ends with a worker review. Two interactive experiments let you inspect probability normalization and stale-queue waste directly.
A useful source, viewed from below the API
Hugging Face's A curriculum for post-training agents, published October 20, 2026, connects supervised traces, distillation, outcome-based RL and environment-based training. Its final section points toward asynchronous generation and weight synchronization. It is a useful route into the broader training methods.
For an inference engineer, the next question is what work each method sends to the serving layer. Existing supervised traces may require no live generation during an update. On-policy distillation asks a student to sample and a teacher to evaluate. Multi-turn rollouts interrupt generation with observations from outside the model. Those are different request shapes, output requirements and resource lifetimes.
The course focuses on those serving consequences. Reward design, agent architecture and optimizer selection remain separate subjects. The original curriculum is linked throughout as a reading source; the exercises and visuals here are newly authored rather than reproductions of its infographic.
A token probability needs a precise meaning
Two processes can report different log probabilities for the same token while using identical weights. One may report the raw model softmax; the other may report the distribution after temperature and filtering. Comparing them without declaring that difference produces an apparent policy change.
Use the inspector below with its initial settings. Switch the saved record to the raw probability. Then remove filtering by retaining all four tokens. The ratio returns to one at temperature one because the normalization domains match again. Lowering temperature introduces another transformation to account for.
Which probability did you store?
Keep the weights fixed. Change the sampler, then compare the probability of token A with the saved rollout record.
Raw model
Before sampling transforms
- A50.0%
- B30.0%
- C15.0%
- D5.0%
Temperature
Softmax at T = 1
- A50.0%
- B30.0%
- C15.0%
- D5.0%
Sampled policy
Keep top 2; normalize again
- A62.5%
- B37.5%
- CExcluded
- DExcluded
Recomputed probability / saved probability
0.6250 / 0.6250This is a synthetic consistency check, with no model update or floating-point noise. The behavior-policy provenance concept defines the full record. The probability-mismatch interview problem turns the discrepancy into an executable investigation.
Token identity matters just as much as the probability field. A tool observation can have zero action loss while remaining necessary context for the next generated token. Reformatting a transcript through another chat template may change token boundaries. The free record lesson makes the prediction-row shift explicit with a small runnable fixture.
Known-token scoring has its own memory budget
Once a student continuation exists, the teacher can evaluate those known token choices with causal masking. It does not have to generate another answer. This connects directly to prefill versus decode: the sequential sampling dependency has disappeared, although the underlying model computation remains.
The scorer's output can become a separate bottleneck. A selected-token probability, a sparse candidate list and a dense vocabulary distribution carry different amounts of information. Reducing the response payload is safe only if the consumer can still compute its intended objective.
The teacher-scoring exercise derives the output buffer from batch, positions, vocabulary and precision, then checks a chunked alternative. It also separates peak storage from total work. A smaller temporary tensor does not by itself prove higher throughput, and averaging chunk means can change the loss when valid-token counts differ.
Keep the queue useful
Ordinary serving goodput often counts requests that meet latency and success requirements. A rollout consumer adds another eligibility condition: the sample may need to come from a sufficiently recent policy. That condition can expire while the sample waits in a perfectly functioning queue.
The experiment below separates generated, consumed, discarded and waiting records. Playback is optional. Try a smaller queue admission limit while keeping trainer capacity fixed, then inspect whether the same useful work required fewer generated samples.
A busy generator can feed a stale queue
Step through admission, freshness checks and training. Then lower the queue limit and compare how many generated samples are discarded.
The queue is empty.
- Generated
- 0
- Consumed
- 0
- Discarded as stale
- 0
- Not started
- 0
This tick: no stale discard. Lag is checked before this tick’s update; waiting labels show the version after it.
Its ticks are abstract. Generation takes no simulated time, samples have equal cost, and stale scanning is free. A real service needs enough outstanding work to cover generation, tools and scoring. The point is to locate waste and preserve the accounting, not to recommend a production queue size.
The stale-queue incident adds consumption-time eligibility and a changed-constraint follow-up. The course asks you to inspect the task mix among discarded records as well. If long episodes are disproportionately removed, a faster consumer feed may represent a different training population.
A weight update changes reusable state
Changing weights while keeping token text intact does not preserve the meaning of cached K and V. A new forward pass must use state compatible with its parameters. Shared prefix entries and external cache copies need the same review as active request-local state.
There is also a historical distinction. Replaying previously generated tokens under new weights rebuilds their cache, but it does not rewrite the probabilities under which they were originally sampled. The existing weight-sync question already develops this engine-level tradeoff; the course uses it as a reference rather than adding a second answer to the same question.
At a release boundary, test both a resumed continuation and a fresh request with a previously cached prefix. One test exercises active state; the other exercises reusable state that may survive beyond any single request.
Follow the code with a specific question
The implementation review for this publication pins TRL to commit e29597af, checked October 29, 2026. Start at the async rollout worker to inspect token records, then the async trainer to find the consumption-time version check. The distillation implementation provides a separate reading assignment on chunked vocabulary projection and loss.
These are inspected source paths, not locally reproduced training results. Keep the question narrow: what information crosses this boundary, and which test would expose its loss? If you want practice following that kind of path first, Reading inference engines develops the method. Start the new course with the generation-and-scoring map, then carry its notebook into the final review.
The questions and concept pages that test what this post covers:
FAQ
It is a sampled attempt whose token history and generation metadata become training input. It may span several model turns and tool observations. The serving contract must preserve the sampled choices and the probabilities and policy identities needed by the consumer.
Discussion (3)
For a source-reading session, follow one generated token from the server response into the collator. Write down every place its position or mask changes. That path is usually a better starting point than reading all of the trainer's configuration fields.
A queue experiment needs an explicit population. Conversations, completed scoring groups and packed training sequences can have different counts. Record the conversion between them before comparing dashboards from two implementations.
The exercises intentionally stop before device signoff. A CPU check can expose a wrong denominator or stale revision, but stream ordering, transfer completion and kernel numerics still need tests on the actual serving stack.