Fast first token.
Useful token stream?
A request can start quickly and still stall during decode. Use this eight-request trace to check both first-token latency and the slowest token gap, then calculate how much output met the entire service objective.
All records are authored teaching data. No GPU, serving engine or customer was measured. The window is fixed at 20 seconds and all eight requests finish inside it; changing a deadline changes the qualifying set, not the traffic.
Which requests qualify?
Before revealing the result, count requests that succeeded and met both deadlines. Equality meets the deadline. Every row belongs to the same completed 20-second window.
| Request | TTFT (ms) | Max gap (ms) | Output tokens | Succeeded |
|---|---|---|---|---|
| A | 120 | 18 | 80 | Yes |
| B | 180 | 24 | 120 | Yes |
| C | 300 | 30 | 60 | Yes |
| D | 310 | 20 | 90 | Yes |
| E | 140 | 75 | 200 | Yes |
| F | 220 | 32 | 110 | Yes |
| G | 90 | 16 | 30 | No |
| H | 500 | 60 | 160 | Yes |
Definitions that make the result reproducible
- TTFT is the elapsed time from client request submission to its first output token. It includes queueing, execution and delivery visible to that client.
- Maximum token gap is the largest interval between consecutive output tokens for one request. It catches a stall that a mean time per output token can hide.
- A qualifying request succeeds and meets both deadlines. This worksheet credits all emitted tokens of qualifying requests and zero tokens of other requests. Name that whole-request rule when quoting the token rate.
- The denominator is the entire 20-second observation window. Counting the failed request’s emitted tokens in raw throughput does not make that request successful.
Worked answer at 300 ms TTFT and 40 ms maximum gap
A, B, C, F qualify. Their tokens sum to 370; 370 / 20 = 18.5 tokens/s from qualifying requests. All rows sum to 850; 850 / 20 = 42.5 tokens/s raw throughput. Request goodput is 4 / 20 = 0.2 requests/s.
E meets the first-token deadline but misses the gap deadline. G has low latency but fails. A throughput number alone hides both problems. These eight rows are too few to estimate a production p99.
Copy the complete input fixture
{
"kind": "synthetic teaching fixture",
"windowSeconds": 20,
"ttftSloMs": 300,
"maxGapSloMs": 40,
"requests": [
{
"id": "A",
"ttftMs": 120,
"maxGapMs": 18,
"tokens": 80,
"success": true
},
{
"id": "B",
"ttftMs": 180,
"maxGapMs": 24,
"tokens": 120,
"success": true
},
{
"id": "C",
"ttftMs": 300,
"maxGapMs": 30,
"tokens": 60,
"success": true
},
{
"id": "D",
"ttftMs": 310,
"maxGapMs": 20,
"tokens": 90,
"success": true
},
{
"id": "E",
"ttftMs": 140,
"maxGapMs": 75,
"tokens": 200,
"success": true
},
{
"id": "F",
"ttftMs": 220,
"maxGapMs": 32,
"tokens": 110,
"success": true
},
{
"id": "G",
"ttftMs": 90,
"maxGapMs": 16,
"tokens": 30,
"success": false
},
{
"id": "H",
"ttftMs": 500,
"maxGapMs": 60,
"tokens": 160,
"success": true
}
]
}Take the worksheet to a real serving run
Record the model and tokenizer revisions, GPU part and count, engine version, precision, prompt/output length distribution, arrival process, concurrency, warmup and test window. Keep failed and cancelled requests in the accounting. Use a monotonic clock within one observer; do not subtract client and server wall-clock timestamps without a synchronization model.
Streaming transports can group several tokens into one delivered chunk. If you only observe chunks, call the metric a chunk gap or retain token-level events before claiming token-gap latency. A client trace alone cannot assign a regression to queueing, prefill or decode; correlate it with server spans and counters.
For a concrete benchmark interface, vLLM v0.32.0’s serving benchmark documents TTFT, TPOT, inter-token latency and goodput options. Check its exact definitions before comparing its output with this whole-request teaching rule.