TL;DR: Goodput is the rate of requests that finish inside both latency targets (TTFT and TPOT), not the rate of tokens produced. For Llama 3.1 8B on one H100 at 2,048 tokens of context, batch 64 decodes at 9.9 ms per step and 6,450 tokens/s; batch 128 makes 8,505 tokens/s (32% more) at 15.1 ms per step, so against a 10 ms TPOT target every request in the batch misses and goodput falls to zero.
How to approach it
Pin the definition down before computing anything: which latency metrics, which percentile, and what fraction of requests must meet them. Ask for the TTFT and TPOT targets, the context-length mix, and whether the target applies per request or per token. Then show the mechanism that makes the two numbers diverge, which is that every sequence in a decode step waits for the same step.
A strong answer
Throughput counts work done. Goodput counts work a user would accept. The DistServe paper (Zhong et al., OSDI 2024) made the term standard for LLM serving: the highest request rate at which a target fraction of requests, say 90%, meet both the TTFT and the TPOT objective. Divide by GPU count and it becomes the number a capacity plan is built on.
The divergence comes from batching. A decode step reads the weights once and every running sequence's KV cache, and all of those sequences receive their next token when the step ends. TPOT is therefore a property of the batch, not of the request. Work it for an 8B model:
Llama 3.1 8B, BF16, one H100 SXM (3.35 TB/s HBM3, 989 TFLOP/s dense BF16)
weights = 8.03e9 params x 2 B = 16.06 GB
KV per token = 2 x 32 layers x 8 kv_heads x 128 x 2 B = 128 KiB
KV per sequence = 2,048 tokens x 128 KiB = 268 MB
step(B) = (16.06 GB + B x 268 MB) / 3.35 TB/s
B = 64 = (16.06 + 17.18) GB / 3.35 TB/s = 9.92 ms
B = 128 = (16.06 + 34.36) GB / 3.35 TB/s = 15.05 ms
compute check = 128 x 2 x 8.03e9 FLOP / 989e12 = 2.08 ms (memory-bound)
largest B at 10 ms TPOT = (33.5 - 16.06) GB / 268 MB = 64.97 -> 64
With 256 output tokens per request, the request rate that a full batch sustains follows from Little's law, batch divided by request duration:
| Batch | Step time | Tokens/s | Meets 10 ms TPOT | Goodput (req/s) |
|---|---|---|---|---|
| 16 | 6.08 ms | 2,633 | yes | 10.3 |
| 32 | 7.36 ms | 4,349 | yes | 17.0 |
| 64 | 9.92 ms | 6,450 | yes | 25.2 |
| 96 | 12.49 ms | 7,688 | no | 0 |
| 128 | 15.05 ms | 8,505 | no | 0 |
Throughput keeps rising past 64 because the weight read is amortized over more sequences. Goodput does not, because the extra sequences make every step slower for everyone already in the batch. In this idealized model the cliff is vertical, since every step is identical. On a real server contexts vary and some steps carry prefill chunks, so SLO attainment falls over a range of load rather than at one point, but the shape is the same: goodput rises, peaks and then drops while throughput is still climbing.
TTFT gives a second route to the same failure. Admitting more requests to raise throughput lengthens the queue in front of prefill, and a larger chunked-prefill budget makes prefill more efficient while stretching the steps that decoding users wait on.
What follows in practice: cap concurrency at the largest batch that meets the target percentile, and add replicas beyond that instead of deepening the batch. Measure it with an open-loop load generator that sweeps the arrival rate, plot SLO attainment against request rate, and read goodput as the highest rate where attainment stays above target. A benchmark that submits every request at time zero measures saturated throughput and says nothing about goodput.
The reversal: with no latency objective (offline generation, evals, embedding backfills), goodput equals throughput and the right batch is the largest the KV pool holds, about 200 sequences of 2,048 tokens here. With a loose objective, such as 50 ms TPOT for a reading-speed chat, the knee moves past KV capacity and throughput becomes the binding number again.
What interviewers probe next
"The target is now p90 TPOT, not every request." Steps vary with context and prefill load, so pick the batch where 90% of steps finish under 10 ms; with a spread of contexts that sits below 64 for some steps and above it for others.
"Would an FP8 KV cache change the answer?" Yes, in this regime: KV per sequence halves to 134 MB, the largest batch under 10 ms becomes (33.5 - 16.06) GB / 134 MB = 130, and goodput roughly doubles.
"How do you measure goodput on a live server?" Per request, from TTFT and that request's mean inter-token latency, counted against the objective each second; token-level averages hide the requests that missed.
How it's scored
Weak: defines goodput as "useful throughput" with no SLO attached, or claims larger batches always help because tokens per second go up.
Hire: defines goodput as requests per second meeting both TTFT and TPOT targets, and explains that a batch shares one step time.
Strong Hire: computes the step time from weight and KV bytes, finds the batch at which TPOT crosses the target, converts it to requests per second, and says when goodput and throughput coincide.
Common mistakes
- Reporting goodput from a closed-loop benchmark at a fixed concurrency, which never builds a queue and so never shows TTFT misses.
- Counting tokens that met the SLO instead of requests. One slow request with 2,000 tokens and many fast short ones can look healthy by token count.
- Leaving KV reads out of the step model. At 2,048 tokens of context the KV of 64 sequences (17.2 GB) outweighs the weights.
- Treating the knee as a fixed engine property. It moves with context length, KV precision and the TPOT target.