Measure arrivals and qualified completions
Record intended arrivals, delivery events and every terminal outcome. Compare configurations by requests that meet the whole contract, with a fixed observation window and explicit censoring.
20 MIN
By InferenceInterviews · Updated
Course lessons Lesson 2 of 24
TL;DR: Count qualified completions against all offered work, using intended arrivals and an explicit observation window; otherwise client backpressure, failures and unfinished requests can manufacture a throughput improvement.
Where you are. You can trace a request. Now you will construct a benchmark report that cannot improve simply by losing difficult requests.
A report with a missing denominator
Two synthetic configurations each receive 100 scheduled requests over ten seconds. A completes 90, with 80 meeting the full contract. B completes 70, with 65 meeting it. B reports 92.9% compliance among completions; A reports 88.9%. Would you release B?
Under an offered-request contract, A qualifies 80% and B 65%. Their qualified completion rates over this specified ten-second window are 8 and 6.5 requests/s. The other requests require explicit terminal or unfinished statuses. They cannot disappear because the server returned an error before producing a token.
Write the acceptance predicate before running the load. Our exercise requires a successful terminal event, first token within 120 ms of intended arrival, and every observed token gap at most 40 ms. A real service may choose other thresholds or a percentile contract, but changing the predicate after observing results invalidates the comparison.
Preserve the load generator's clock
A closed-loop client sends a new request when an earlier request finishes. When the server slows down, the client reduces offered load. That is a useful model of a fixed user population, but it does not reproduce an independent arrival process.
An open-loop experiment schedules arrivals independently of completions. Record intended arrival and actual send. If the generator falls behind, measuring only send-to-first-token hides part of the waiting time. Report that generator lag separately; sufficiently large lag means the intended workload was not delivered faithfully and the run needs correction, not a favorable interpretation.
Execute the acceptance rule
These four records are synthetic, with times in milliseconds on one client clock. The function treats a one-token successful request as having no observed inter-token gap; that is a policy choice made explicit, not a measured gap of zero.
records = [
dict(arrival=0, sent=0, receipts=[80, 110, 145], ok=True),
dict(arrival=100, sent=180, receipts=[230, 250], ok=True),
dict(arrival=200, sent=200, receipts=[270, 320], ok=True),
dict(arrival=300, sent=300, receipts=[360], ok=False),
]
def qualifies(r, ttft_limit=120, gap_limit=40):
ts = r["receipts"]
if not r["ok"] or not ts:
return False
assert all(a <= b for a, b in zip(ts, ts[1:]))
return (ts[0] - r["arrival"] <= ttft_limit
and all(b - a <= gap_limit for a, b in zip(ts, ts[1:])))
assert [qualifies(r) for r in records] == [True, False, False, False]
assert records[1]["receipts"][0] - records[1]["sent"] == 50
assert records[1]["receipts"][0] - records[1]["arrival"] == 130
The second request looks fast from send: 50 ms. Its intended-arrival TTFT is 130 ms, which misses the 120 ms limit. The third has acceptable TTFT but a 50 ms gap. The fourth delivers something and still fails because its terminal outcome is unsuccessful. One request qualifies. With a declared one-second observation window containing all four terminal outcomes, goodput is 1 request/s.
Do not infer token-level ITL from network chunks containing multiple tokens. In that case label the measured quantity chunk gap, retain chunk token counts, and use engine-side timing for production gaps. The code assumes one receipt per token only for this exercise.
Keep the window and uncertainty visible
A ten-second arrival window followed by a drain is not automatically a ten-second completion-rate window. Publish both durations and distinguish cohort attainment from completions per wall-clock second. Requests still running at cutoff are censored; preserve their age and status rather than assigning them zero latency. A retry adds an attempt and consumes resources even if the logical request eventually succeeds.
Sweep offered load with the same prompt/output distributions, cache conditions and quality gate. Repeat configurations in interleaved order to reduce drift. Keep raw request records and summarize uncertainty across independent runs; thousands of correlated tokens from one request are not thousands of independent latency samples. Report failure rate and the worst relevant workload slices beside headline percentiles.
Do this before moving on
Change the second record's intended arrival to 120 ms and the third record's last receipt to 310 ms. Predict the new qualifying set before executing the code. Then write a report for a run that ends with two unfinished requests: specify how you will report attainment, latency and completion rate without silently treating either request as successful.
Use the goodput experiment to change the acceptance threshold. Explain why the winning configuration can change without either kernel changing.
Go deeper
-
Interpret zero SLO violations derives what a finite independent sample can establish about a tail requirement.
-
Inference benchmark methodology develops the arrival-clock and censored-request accounting in detail.
-
Streaming chunks hide token latency tests what can actually be inferred from delivery events.
-
Goodput versus throughput connects the request predicate to a capacity decision.
-
Latency decomposition helps locate the delay after the request fails its predicate.
Key takeaways
- Fix the denominator and acceptance predicate before collecting results.
- Retain intended arrivals, actual sends and terminal outcomes.
- Distinguish cohort attainment from a rate over elapsed time.
- Preserve uncertainty and failures beside the headline throughput.
Check yourself
Work it out before revealing the explanation. Include the units, assumptions and the condition that would change your answer.
1After the two exercise edits, which original records qualify?
2B completes 70 of 100 offered requests and 65 meet the contract. What is offered-request attainment?
3A one-token successful request has no later receipt. What is its observed inter-token latency?
4A ten-second arrival cohort drains for five more seconds. May all completions be divided by ten and labeled wall-clock completion throughput?
Sign in to track which lessons you have finished.