Why does prefill turn compute-bound after a few hundred tokens on an H100, and how long does a 4K-token prompt take on a 70B model?

TL;DR: A prefill GEMM over $P$ tokens reuses each weight $P$ times, so in BF16 its intensity is about $P$ FLOP/byte and it crosses the H100 SXM's ridge of 295 FLOP/byte at roughly 300 tokens. A 4,096-token prompt on Llama 3.3 70B is about 591 TFLOP, which is 0.30 s on two tensor-parallel H100s at dense BF16 peak and about 0.5 s if the GEMMs sustain 60% of it.

How to approach it

Frame prefill as a matrix-matrix product, a batch of $P$ token vectors against each weight matrix, and compute its intensity the same way you would for decode. Then ask what changes the number: weight and activation precision, tensor-parallel degree, prompt length (attention is quadratic), and whether chunked prefill splits the prompt across several engine steps. Only then convert FLOPs to seconds.

A strong answer

For a weight of shape $d \times k$ and $P$ tokens, the GEMM does $2Pdk$ FLOPs and moves the weight once plus the input and output activations:

$$I(P) = \frac{2 P d k}{b\,(d k + P(d + k))} \approx \frac{2P}{b} \quad \text{when } P \ll d$$

In BF16 ($b = 2$) that is about $P$ FLOP/byte. The H100 SXM delivers 989 TFLOP/s dense BF16 from 3.35 TB/s of HBM, a ridge of 295 FLOP/byte, so the crossover is a few hundred tokens:

ridge (H100 SXM, BF16 dense) = 989e12 / 3.35e12 = 295 FLOP/byte
MLP up-proj 8192 x 28672:  I(256) = 246   I(310) = 295   I(4096) = 2,493
square 8192 x 8192:        crossover at P = 319
K/V proj 8192 x 1024:      crossover at P = 437 (narrow output, more activation traffic)
FP8 weights and FP8 math: ridge 1979 / 3.35 = 591, I = 2P/1, crossover still ~300

FP8 does not move the crossover, because halving the bytes doubles the intensity and doubling the peak doubles the ridge. It halves the time on both sides of it.

Now the 4K prompt. The weight GEMMs cost $2N_{\text{matmul}}$ per token, and causal attention averages half the prompt as context:

N_matmul (Llama 3.3 70B, excl. input embedding) = 69.50e9
weight FLOPs     = 2 x 69.50e9 x 4,096                   = 569.3 TFLOP
attention FLOPs  = 4 x 80 x 64 x 128 x 4,096^2 / 2       =  22.0 TFLOP
total            = 591.3 TFLOP
TP=2, per GPU    = 295.7 TFLOP / 989e12                  = 0.299 s at peak
at 60% sustained = 0.50 s   (assumed efficiency, measure yours)
weight reads     = 70.55 GB / 3.35e12                    = 21 ms   (7% of the peak time)
sanity: TP=4 halves it to 0.15 s; FP8 GEMMs at TP=2 also give 0.15 s at peak

Prefill at TP=2 also all-reduces the hidden state twice per layer, and those payloads are now large: 4,096 tokens × 8,192 × 2 B is 67 MB per all-reduce, and with ring all-reduce each GPU sends 2.6 MB per token across 160 calls, 10.7 GB for the prompt. At roughly 450 GB/s per direction (half of the H100's 900 GB/s bidirectional NVLink figure) that is 24 ms, 8% of the compute time if not overlapped. This is why prefill benefits from compute-communication overlap while decode does not.

The 0.30 s is the TTFT floor for an otherwise empty server. In production the scheduler decides how much of it a request gets: with chunked prefill (on by default in vLLM v0.31, checked October 2026), the prompt is split into chunks that share each step's token budget with running decodes, so a 4K prompt may take several steps and TTFT grows by the decode work interleaved with it.

The reversal condition has two ends. Below about 300 tokens, a short prompt costs the same 21 ms of weight reads as a decode step, so batching several small prefills together is free throughput. At very long prompts the quadratic attention term takes over: at 131,072 tokens attention is 22.5 PFLOP against 18.2 PFLOP for the weights, 55% of the total, and the attention kernel's efficiency, not the GEMMs', sets TTFT.

What interviewers probe next

"So chunk size 512 is enough to stay compute-bound?" For the weight GEMMs, yes, a 512-token chunk sits at about 474 FLOP/byte. The cost is that every chunk rereads the weights and the prompt's KV so far, so tiny chunks give back efficiency.

"Why is measured prefill below 60% of peak on a short prompt?" At 256 tokens the GEMMs are still at the ridge, tile quantization wastes part of the last tile, and fixed per-layer launches and all-reduce latency are a larger fraction of a 20 ms step.

"What does a B200 change?" Dense BF16 rises to 2,250 TFLOP/s and bandwidth to 8 TB/s, a ridge of 281. The crossover stays near 300 tokens; the 4K prefill drops 2.3x to about 0.13 s at TP=2 peak.

How it's scored

Weak: says prefill is compute-bound "because it is parallel", or prices the 4K prompt by bytes and gets 21 ms.

Hire: derives intensity of about $P$ FLOP/byte, places the crossover near 300 tokens against a 295 ridge, and gets about 0.3 s at TP=2 from 591 TFLOP.

Strong Hire: shows FP8 leaves the crossover in place, adds the causal $T/2$ attention term and its takeover at long prompts, and connects chunked prefill to why measured TTFT exceeds the floor.

Common mistakes

• Using the 1,979 TFLOP/s sparse BF16 figure, which halves the predicted TTFT and puts the ridge at 590.
• Counting attention as full $P^2$ rather than the causal half, overstating it 2x.
• Assuming the crossover is where prefill becomes efficient. At the ridge, memory time equals compute time, so any imperfect overlap between the two costs up to half the peak. Real efficiency needs several times the ridge.
• Quoting TTFT without the queueing and chunk scheduling in front of it, which is often larger than the compute.