Skip to content

Trace a request from tokens to observable output

A serving contract binds token positions and model state to observable deadlines. Trace shapes through prefill and decode before choosing a worker or optimizing its kernels.

18 MIN

By InferenceInterviews · Updated

Course lessons Lesson 1 of 24

TL;DR: Trace positions, tensor shapes and observable events together: prefill consumes the prompt, sampling chooses a token, and the next forward pass incorporates that token into persistent model state.

Where you are. Start here if you can multiply matrices but have not traced a streamed generation through a worker. You will produce a one-request execution ledger.

Predict which token has state

A synthetic decoder receives four prompt tokens. Its first forward pass returns logits, and a sampler chooses token u. Pause before the next forward pass. Does the cache already contain the key and value for u?

It does not. The logits at the final prompt position predict the next token. Sampling produces an identifier; computing that identifier's per-layer state requires another forward pass. Confusing these events causes off-by-one errors in cache lengths, rollback and usage counters. A response can contain a token whose own KV has never been computed, particularly when generation stops immediately after sampling it.

This distinction also explains two phases of autoregressive inference. Prefill evaluates several known positions together, respecting their causal dependencies through a mask. Decode evaluates newly known positions using saved state. Weight matrices are reused in both phases. The amount of independent work changes.

Put dimensions on the path

Use a deliberately small decoder: model width 32, four query heads, two KV heads, head dimension 8, two layers and a vocabulary of 64 tokens. These are exercise inputs, not a named model. Batch size is one. Ignore biases and positional transformations for this shape ledger.

The embedding lookup produces [1, 4, 32]. A query projection maps width 32 to 4 × 8 = 32; each key and value projection maps it to 2 × 8 = 16. After reshaping, queries are [1, 4, 4, 8] in batch, position, head, channel order; keys and values are [1, 4, 2, 8]. Each KV head serves two query heads. Repeating KV physically is an implementation choice, not a requirement of grouped-query attention.

For each query head, attention compares a query with allowed keys, normalizes the scores, and combines the corresponding values. The head outputs concatenate back to width 32. Residual paths and the feed-forward sublayer preserve the position axis. A final projection produces vocabulary logits. A serving implementation may project only the positions whose logits are needed; materializing [1, 4, 64] is not mandatory.

rendering diagram…

The split after sampling is intentional. Emission and model execution are different paths; a slow consumer can delay delivery after the engine has already produced more tokens.

Turn the trace into an observable contract

Record prompt length, generated token IDs, computed positions and emitted events separately. A worker's computation timestamps describe execution. Client receipt timestamps describe the experience, including buffering and transport. Neither can replace the other.

For this exercise, let scheduled arrival be 0 ms, server admission 12 ms, sampling of the first token 38 ms, and client receipt 47 ms. Client-observed time to first token is 47 ms. Admission wait is 12 ms. The remaining 35 ms includes execution and delivery; it cannot all be labeled model compute. If an instrumentation span attributes only 21 ms to device execution, there are 14 ms left to locate in tokenization, host scheduling, serialization or transport. Keep clock domains separate unless you have a synchronization method.

The contract also needs a terminal condition. Specify maximum output length, stop handling, timeout, cancellation and whether a request that returns partial text counts as success. These choices affect both cache lifetime and benchmark denominators. A fixed random seed alone does not promise identical strings across different floating-point kernels or batch shapes.

Do this before moving on

Make a table for a three-token prompt followed by two sampled tokens, stopping immediately after the second sample. Include known tokens, computed positions and emitted tokens after each event. Then repeat with a client that receives both generated tokens in one chunk. Identify which engine facts remain observable at the client.

Calculate the exercise model's KV payload in two-byte elements after the prompt and after consuming u: 2 × layers × KV heads × head dimension × bytes × positions. State what this omits. Revisit the ledger tomorrow without the diagram; reconstruct why sampling cannot advance the computed-position counter.

Go deeper

Key takeaways

  • Sampling chooses an identifier; a later forward pass computes its KV.
  • Preserve token positions and axis meanings when tracing shapes.
  • Client delivery and device execution need distinct instrumentation.
  • Define terminal outcomes before counting successful requests.

Check yourself

Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.

  1. 1After a three-token prompt and two sampled output tokens, stopping immediately after sample two, how many positions have computed KV?

  2. 2What is the tiny model's KV payload after four and five computed positions?

  3. 3A client receives two tokens in one chunk. Can it reconstruct their individual production gaps?

  4. 4The device span is 21 ms and arrival-to-receipt is 47 ms. Is device optimization guaranteed to remove the other 26 ms?

Sign in to track which lessons you have finished.