Follow one token through five ownership boundaries

TL;DR: Trace one pending token through admission, batch preparation, forward execution, sampling and publication. At every boundary, record who owns the request and which event makes its state safe to read or reuse.

Where you are. The first lesson builds the state map that later correctness experiments will try to break.

Start from an observable result

Choose the function that sends a generated token to the caller. Work backward until you can identify the sampled tensor row, the invocation that produced it and the request that owned that row. This gives you a finite path through the repository. Reading every utility before finding that path makes it difficult to distinguish machinery used on every token from machinery used only at startup.

Use one synthetic request with a four-token prompt and a two-token output allowance. Before the first forward pass, the prompt is known but none of its KV has been computed. After prefill, the four prompt positions have KV and the sampler can select the first output token. That sampled token is part of the visible history, but its own KV is usually computed on the next pass. Mark that distinction on the page now; it will explain the speculative-commit failure later.

The return edge carries a token that has been selected but not necessarily processed. It does not mean every state field can advance at the same time.

Give each boundary a contract

Your map should answer concrete questions. Does admission allocate physical pages immediately, or only reserve capacity? Does batch preparation copy the block table, retain it by reference or guarantee that nobody mutates it? Which completion event tells the host that sampled IDs are readable? Can output publication happen after cancellation? Who releases a request's storage?

Separate evidence from guesses. A function named finish might mean the forward pass completed, the request reached EOS or a transport finished sending bytes. Follow the field it updates and the readers of that field. A useful note identifies the symbol and the transition it performs. A guess stays labeled until another call site resolves it.

For the four-token prompt, write two counters after the first sample: known history length five and target-computed prefix length four. If your chosen implementation uses a different convention, translate both counters into actual token positions. Names can differ; the read boundary must still be unambiguous.

Add asynchronous work only after the serial path

An engine may schedule the next batch while the GPU executes the current one. The two batches can have different membership and ordering. Draw separate launch records before merging the timelines. A mutable global list called running cannot explain which request owned a row in a previous invocation.

The same discipline applies to memory. Removing a request from the scheduler prevents future launches, but an older invocation can still own a read or write. Record the completion dependency beside the free operation. If there are multiple consumers, identify all of them before concluding that a page can be reused.

Do this before moving on

Write a five-row map for the synthetic request: boundary, owner, token positions known, token positions computed, and completion evidence. Then add cancellation immediately after launch. Circle the operations that must stop and underline the operation that must still complete before reclamation.

A satisfactory map keeps the launched page alive until the device use finishes, suppresses cancelled output, and keeps the first sampled output separate from computed KV. Use the dated source pointer to compare that map with a real engine; record differences instead of forcing its fields into our naming scheme.

Go deeper

• Iteration-level scheduling explains why batch membership changes between successive invocations.
• Paged KV memory supplies the storage vocabulary needed for the ownership column.
• Cancellation with in-flight ownership turns your completion dependencies into an executable lifecycle model.

Key takeaways

• Follow a visible output backward to its owning invocation.
• Known token history and computed KV can end at different positions.
• Removing a request from scheduling does not complete its outstanding device work.

Check yourself

Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.

1. 1A four-token prompt has completed prefill and one output token was sampled. Must the target already hold five KV positions?

   Yes, because five token IDs are knownNo, the sampled output may be the next pending input

2. 2Which event proves a cancelled page is reusable?

   Answer it, then reveal