TL;DR: 1,000 tokens per second per user means a 1 ms decode step, but an H200 needs 14.7 ms just to stream Llama 3.3 70B's FP8 weights at 4.8 TB/s, and 7.3 ms even at INT4. Without speculation the claim is 7x to 15x beyond the bandwidth floor; it matches almost exactly the aggregate throughput of 16 concurrent users in FP8, which is the likeliest reading.
How to approach it
Convert the claim into a step time and compare it to bytes over bandwidth, before discussing anything else. Then list the conditions that would change the arithmetic and ask the vendor for each: is it per user or aggregate, what weight precision, is speculative decoding on and with what acceptance on what data, which context and output lengths, and does "tokens" include the prompt. The floor turns a debate into a short list of questions.
A strong answer
Per-user throughput at decode is one over the step time, and a batch-1 step must read every weight once. On one H200 SXM (141 GB, 4.8 TB/s):
claim = 1,000 tok/s per user -> step = 1.0 ms
Llama 3.3 70B = 70.55e9 params
BF16 weights = 141.1 GB -> does not fit in 141 GB, before any KV cache
FP8 weights = 70.6 GB -> 70.55e9 / 4.8e12 = 14.7 ms -> 68 tok/s
INT4 weights = 35.3 GB -> 7.35 ms -> 136 tok/s
bandwidth needed for 1 ms at FP8 = 70.6 TB/s = 14.7x an H200
sanity: compute is not the limit; 2 x 70.55e9 FLOPs at 1,979 TFLOP/s dense FP8 = 0.07 ms
So at batch 1 the claim is off by 7.4x at INT4 and 14.7x at FP8. No kernel tuning closes that, because the floor assumes 100% of datasheet bandwidth and real decode kernels run below it.
Now find the reading that makes it true. The roofline bound from decodeStep for FP8 weights and an FP8 KV cache at 2,048 tokens of context:
batch 16: step = 15.8 ms -> aggregate 1,012 tok/s, per user 63 tok/s
batch 32: step = 16.9 ms -> aggregate 1,890 tok/s, per user 59 tok/s
sanity: the step barely grows with batch because weight reads dominate,
which is why "1,000 tok/s" falls out of 16 users almost exactly
| Hypothesis | What it requires | Plausible? |
|---|---|---|
| Aggregate, not per user | 16 users at FP8, 63 tok/s each | Yes, at the floor; at 80% MBU it needs about 20 users |
| Counts prompt tokens | FP8 prefill peaks near 14,000 tok/s (1,979e12 / 139e9 FLOPs per token) | Yes, and inflates any blend |
| Speculative decoding | At least 7.4 tokens per verified step at the INT4 floor, with free drafting | Only on near-copy workloads |
| Different model | An 8B in INT4 has a 0.84 ms floor, about 1,195 tok/s | Yes, but it is not a 70B |
| Different hardware | Weights held in on-chip SRAM across many chips | Yes, but not "one H200" |
The speculation row deserves its number. With per-token acceptance and draft length , expected tokens per verification step is . At and that is 6.1 tokens, so even with a free draft model the INT4 floor yields tok/s. Reaching 1,000 needs acceptance above 0.9 at long drafts, which happens on code edits and templated output where ngram or suffix drafting copies the prompt, not on open chat. Ask for the acceptance rate and the dataset.
What to send back is specific: per-user tokens per second at batch 1 and at the stated concurrency, the weight and KV precision, the speculative method and its measured acceptance on your traffic, and the output length distribution. A vendor who answers with a single number for all of these is reporting the best case.
The reversal condition: if the vendor means aggregate throughput at modest concurrency, or the workload is dominated by copy-heavy edits where speculation accepts 8 or more tokens per step, the claim is consistent with physics and the conversation becomes about price per million tokens and the per-user latency you actually get.
What interviewers probe next
"They say it is per user, with speculative decoding. What do you measure?" Per-request inter-token latency on your own prompts at the concurrency you will run, plus the engine's reported acceptance rate. Speculation gains shrink as batch grows, because verification of many sequences starts to cost compute.
"Would two H200s at TP=2 make it real?" The FP8 floor drops to 7.35 ms, 136 tok/s, still 7x short, and 160 all-reduces per token add a latency term that does not shrink with GPU count.
"Same claim on a B200?" 8 TB/s and NVFP4 weights of 39.7 GB give a 4.96 ms floor, about 200 tok/s at batch 1. Closer, still 5x short without speculation.
How it's scored
Weak: divides FLOPs by the H200's TFLOP/s, concludes the claim is easy, or argues from "the vendor is probably optimizing a lot".
Hire: converts the claim to a 1 ms step, shows the 14.7 ms FP8 floor, and identifies aggregate throughput at about 16 users as the consistent reading.
Strong Hire: quantifies the speculative-decoding escape with expected tokens per step, notes BF16 does not even fit, and turns the analysis into the four numbers to request from the vendor.
Common mistakes
- Using H100 bandwidth (3.35 TB/s) for an H200. The H200's 4.8 TB/s is the whole reason it is in the claim.
- Forgetting that the floor is a ceiling on performance at 100% bandwidth, so any measured number above it is a measurement or definition problem.
- Comparing to "tokens per second" from a benchmark that mixes input and output tokens.
- Assuming speculative speedups measured at batch 1 hold at the vendor's concurrency.