Turn a latency target into a request contract

TL;DR: Define first-token and token-gap deadlines at the client, then preserve enough worker capacity to meet both. In this course the synthetic contract is 300 ms to the first token and at most 30 ms between consecutive tokens.

Where you are. This is the first lesson: write the contract that the later memory and scheduling policies must satisfy.

Decide what the clock observes

Start the first-token clock when the client sends the request. Stop it when the client receives the first output token. That interval includes transport, queueing, prompt processing and delivery. An engine timer around its forward pass covers only part of that path. Keep both measurements, but give them different names.

For subsequent tokens, this course uses the maximum gap between consecutive tokens received by the client. One late gap disqualifies the request. This is a stronger condition than a limit on average time per output token. A stream that pauses for a long interval and then emits several buffered tokens can pass an average while failing the gap contract.

The 300 ms and 30 ms limits are authored exercise inputs. They are not industry defaults or claims about a particular product. A voice application, an interactive editor and a background generation job can choose different contracts. Change the limits before selecting a scheduler; a scheduler optimized for the wrong contract can look excellent in its own benchmark.

The diagram separates queue time from execution. If the first token is late, a kernel optimization helps only the execution portion it actually shortens. If tokens arrive in bursts, compare server completion timestamps with client receipt timestamps before blaming the decoder.

Write the failure policy now

A request qualifies only if it completes successfully and meets both limits. An error after several output tokens is a failed request. Keep its emitted tokens in the observed output total, and keep the request in the offered-work denominator. Otherwise a change that fails expensive requests can appear to improve performance.

Cancellations need an explicit policy too. In the course benchmark, a client cancellation is recorded separately and does not count as a successful completion. A production product may distinguish a voluntary stop after a satisfactory answer from a timeout cancellation. Preserve the reason so the analysis can make that distinction later; silently deleting either class makes comparisons unreliable.

Use a fixed observation window. If a request begins inside it and finishes outside, decide whether to drain the cohort or report it as unfinished. Do not change that rule between candidates. The small trace in the goodput lab avoids this boundary ambiguity by placing every terminal event inside its twenty-second window.

Connect the contract to the worker

The scheduler has two kinds of debt. A waiting prompt needs enough work to reach its first token. An active sequence needs its next decode step before its token-gap budget expires. Memory admission creates another obligation: the request must have somewhere to put the KV state produced by that work.

Your design sheet should record who owns each obligation, when it starts and which metric exposes a miss. A global requests-per-second target cannot replace this sheet because requests can contain very different prompt and output lengths. Keep the length distribution with the target.

Do this before moving on

Open the SLO goodput lab. Before revealing any reasoning, identify the requests that meet the default contract. Then relax only the token-gap threshold. Explain why the observed output rate stays fixed even though qualified output rises.

A satisfactory response names A, B and C at the default limits, excludes G because it fails, and keeps the twenty-second denominator. A stronger response explains why this eight-request fixture cannot estimate a production tail percentile. Record those points beside your contract.

Go deeper

• Latency decomposition separates the intervals you will instrument before selecting a fix.
• Benchmark methodology adds workload controls needed for a defensible comparison.
• Capacity planning in tokens translates the request mix into separate phase demands.

Key takeaways

• Choose the observation point before interpreting a latency number.
• A maximum token-gap limit and an average output-time limit grade different behavior.
• Failures remain visible in the offered workload and emitted-token totals.

Check yourself

Say or write your answer first, numbers included, then reveal. Producing it from memory is the skill an interviewer is checking.

1. 1A request has a 120 ms first token and one 75 ms gap. Does it meet the course contract?

   Yes, because its first token is earlyNo, because every gap must be at most 30 ms

2. 2What evidence distinguishes a decoder stall from buffered delivery?

   Answer it, then reveal