Map generation and scoring before sizing the worker
Distinguish model sampling, known-token scoring, environment waiting and optimization. Build a workload map whose counters preserve what each stage actually produces.
12 MIN
By InferenceInterviews · Updated
Course lessons Lesson 1 of 6
TL;DR: A post-training workload alternates between choosing unknown tokens and evaluating known ones. Label those dependencies before sizing capacity: student generation is sequential, teacher scoring can evaluate a known continuation causally, and a tool wait supplies no runnable GPU work by itself.
Where you are. Start a review notebook for a synthetic rollout service. Your first artifact is a map of the work, its owner and its output.
Before you read. Two conversations each spend a long interval waiting for a tool. Does doubling that waiting time necessarily double the GPU work needed to finish them? Which missing fact changes the capacity decision? Write a prediction and the reason for it; return to it after the worked example.
Learning objective: produce a workload ledger whose counters can be reconciled after a turn is replayed, without treating tool observations as model actions.
Follow one continuation
A rollout is an attempted solution sampled from a model. It can be a single answer or a conversation with tool calls and observations. The trainer uses the resulting record to update parameters. Those updates eventually return to the inference engine, which produces later rollouts from a different policy.
Consider a student producing tokens A, B and C. It must select A before B can be conditioned on A. Once the whole sequence exists, a teacher can evaluate the probability of each observed choice under causal masking. The teacher receives known token IDs; it does not need to sample a replacement answer.
Causal evaluation still prevents a position from seeing its future. Parallel evaluation of known positions is compatible with that constraint, just as it is during prefill. The diagram establishes a dependency difference, not a latency ratio. Model size, context, output format and implementation determine the actual cost.
Translate the training method into requests
The method name tells you where the target comes from. Your serving map needs to go further and state what a worker is asked to return.
| Method or setting | Inference work to identify | Output needed downstream |
|---|---|---|
| Supervised traces | Offline trace generation, if traces must be produced | Tokenized examples and selected loss positions |
| On-policy distillation | Student sampling followed by teacher evaluation | Student trajectory and the specified teacher signal |
| Outcome-based RL | Repeated sampling for a prompt | Generated choices, behavior probabilities and scores |
| Multi-turn environment | Generation interleaved with tool waiting | Turn boundaries, observations and generated-token masks |
| Asynchronous execution | Generation overlapping training | Versioned records, bounded queues and explicit update boundaries |
Supervised training on an existing dataset need not invoke a rollout engine during each update. Distillation can require much richer teacher outputs than one reward per answer. RL can use a learned reward model, a verifier or another scoring mechanism; it does not universally eliminate additional models. These differences change the serving budget before any kernel is selected.
Count the right population
Suppose an illustrative conversation contains 12 prompt tokens, eight generated action tokens and five tool-observation tokens. Its final record has 12 + 8 + 5 = 25 positions, of which eight are model actions. Those are input counts, not measured tokenization of a real conversation.
If a new turn resends earlier context, the engine may process some positions again or reuse a compatible prefix. The number of forwarded positions therefore need not equal 25. The trained-token count can differ again because an objective may exclude padding, truncated answers or other positions. Keep generated, forwarded and trained counters separate, with a written definition for each.
A slow tool expands conversation wall time without necessarily occupying a decode slot. Whether its KV stays resident depends on the engine and request lifecycle. Record waiting conversations separately from runnable inference requests; otherwise an increase in agent concurrency looks like an increase in GPU demand even when the GPU is starved.
Reconcile a replayed turn. Split the illustrative eight generated actions into two turns of four. The first has the 12-token prompt; the second receives the original prompt, four previous actions and five tool-observation tokens. Assume each turn starts a fresh request with no cache reuse. Count forwarded input positions separately from newly sampled outputs.
first-turn prefill positions = 12
second-turn prefill positions = 12 + 4 + 5 = 21
total prefill positions processed = 12 + 21 = 33
newly generated actions = 4 + 4 = 8
final unique record positions = 12 + 4 + 5 + 4 = 25
sanity: 33 counts replayed input; 25 counts the final record
The 33 positions count prefill only; decode forwards are outside that counter. The final sampled token of a turn may never be forwarded before the request ends. A counter called simply "tokens processed" would conceal both choices.
| Proposed shortcut | What it loses | Evidence to retain |
|---|---|---|
| Charge GPU work from 25 final positions | Repeated context processing | Per-invocation input positions and cache hits |
| Count all positions as model actions | Prompt and tool tokens become actions | Origin mask for each recorded position |
| Charge all waiting conversations as decoders | Waiting has no ready forward pass | Runnable state and resident KV separately |
Now allow the second turn to reuse exactly 16 computed prefix positions. It has 21 - 16 = 5 uncached prefill positions, so the two turns process 12 + 5 = 17 prefill positions. The number of model actions remains eight. That is a conditional accounting result: the previous turn must actually have computed all 16 positions under a compatible model and cache identity. Merely possessing their token IDs does not establish a cache hit.
Do this before moving on
Draw the path for one prompt with two sampled attempts and one tool call per attempt. Label which component owns generation, tool execution, scoring, the ready queue and the parameter update. For each arrow, write the record it transfers and the counter that measures it.
Change the setup to an existing supervised dataset. Cross out every service that is no longer needed during the update loop. Your map passes when the remaining requests describe actual work, and when a tool-waiting conversation cannot be mistaken for a runnable decode request.
Go deeper
- Prefill versus decode explains the dependency and arithmetic differences behind known-token evaluation.
- Iteration-level scheduling shows how available generation requests fill a changing batch.
- Behavior-policy provenance defines the record required when those outputs become training input.
Key takeaways
- Separate choosing a token from evaluating a known token.
- A tool wait is a conversation state, not proof of GPU work.
- Generated, forwarded and trained tokens count different populations.
Check yourself
Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.
1A teacher is given the student's complete token sequence. Must it sample another continuation to score those choices?
2The illustrative record has 12 prompt, 8 generated and 5 tool tokens. How many positions and model actions does it contain?
3In the replay fixture, the first turn computed only 15 reusable prefix positions, although 16 token IDs are known. How many uncached prefill positions remain on turn two, and does the action count change?
Sign in to track which lessons you have finished.