Change the inputs.
Understand the tradeoff.
Eight interactive experiments in inference engineering. Make a prediction, move a constraint, and follow the arithmetic. Every result is a model with explicit limits.
A bigger batch. A different bottleneck.
- Ideal step time
- 7.04ms
Sequential linear + attention phases
- Aggregate ceiling
- 1,136tok/s
142 tok/s per sequence
- Linear bottleneck
- HBM
Attention adds 2.56 ms
Hollow markers exceed the chosen memory budget. The curve describes ideal phase costs, not measured performance.
Show the calculation
Linear weight reads = 15.01 GB / GPU
Linear FLOPs = 120.07 GFLOP / GPU
HBM time = 4.48 ms
Linear compute time = 0.12 ms
Linear phase = max(HBM, compute) = 4.48 ms
KV reads = 8 × 8,192 × 131,072 B / 1
= 8.59 GB / GPU
Attention FLOPs = 34.36 GFLOP / GPU
Attention phase = max(KV / bandwidth, FLOPs / BF16 peak)
= 2.56 ms
Step floor = 4.48 + 2.56 = 7.04 ms
Aggregate ceiling = 8 / 0.007045 s = 1,136 tok/s43.33 GB remains inside the chosen budget.
If you double the batch, does each person get their next token sooner?
Work through the decode derivationModel boundaries and primary sources
Decode and memory: dense Llama GQA with evenly sharded KV heads (TP 1, 2, 4 or 8). Linear matrix weights exclude input embedding lookups and norms. The latency model adds sequential linear and attention phase floors. It excludes launch gaps, communication, sampling, softmax overhead, activation traffic, KV writes and cache effects. FP8 is idealized native linear arithmetic; scale overhead, mixed-precision exceptions and quality loss are not modeled. Memory estimates include all model parameters at a uniform precision, BF16 KV, page rounding and a user-selected reservation.
Serving experiments: prefill deadlines use a specified linear cost curve. Prefix ownership counts full shared blocks and private tails. Goodput applies a request-level contract to an authored trace. All three are synthetic and do not reproduce a particular engine release. See the vLLM v0.31.0 scheduling discussion and prefix-cache design for implementation context.
Context growth: the exact pair count is for dense causal attention. Its time coefficients are synthetic, cover all layers and heads, and assume serial phases. DeepInfer’s FLOPs estimator, checked October 2026, is a source-reading assignment in context-sensitive accounting. It also handles compressed and windowed attention; this lab does not reproduce that model or its performance.
Hardware: dense vendor peaks, not achieved performance. Reference data checked 2026-10-16. The 90% budget and 4 GB reserve are illustrative inputs, not vLLM defaults. GB means 10⁹ bytes; KiB means 1,024 bytes.