TL;DR: First compute what FP8 should have bought: for Llama 3.3 70B on TP=4 H100s at batch 64 and 4K context, the bandwidth-bound decode step should drop from 16.9 ms to 11.7 ms, about 1.45x more throughput. A 15% loss means the real FP8 step is near 19.9 ms, 1.7x the expectation, which is too large for scale overhead alone (about 1 ms). One fixed-batch profiler trace per precision splits it: GEMM time up means a slow or fallback kernel, GEMM time down with the step up means extra kernels or a config change.
How to approach it
Pin down the measurement before the cause: which GPU (Ampere, Hopper and Blackwell take different FP8 paths), weights only or weights and KV cache, static or dynamic activation scales, and whether "throughput" means output tokens per second or requests per second. Then ask for the regime of the loss, decode or prefill, because FP8 helps the two for different reasons: decode through halved bytes, prefill through a 2x dense tensor peak (989 to 1,979 TFLOP/s on H100 SXM).
A strong answer
Set the expectation with the decode floor per GPU:
Llama 3.3 70B, TP=4, H100 SXM 3.35 TB/s, batch 64, 4,096 context, BF16 KV
weights BF16 = 70.55e9 x 2 B / 4 = 35.28 GB -> 10.53 ms
weights FP8 = 70.55e9 x 1 B / 4 = 17.64 GB -> 5.27 ms
KV reads = 64 x 4,096 x 327,680 B / 4 = 21.47 GB -> 6.41 ms
BF16 step 16.94 ms, FP8 step 11.68 ms -> expected throughput gain 1.45x
observed 15% loss -> FP8 step ~ 16.94 / 0.85 = 19.93 ms, 1.71x the expectation
The 8.3 ms gap is the thing to explain. Three candidate sources, ranked:
- The kernel path. FP8 GEMMs must hit a native tensor-core kernel. On Ampere there is no FP8 tensor-core math, so engines run weight-only FP8 kernels that dequantize to BF16 inside the GEMM; at batch 64 and above that dequantize work competes with the math and can lose to plain BF16 cuBLAS. On Hopper, block-scaled checkpoints (128x128 weight blocks with FP32 scales, the DeepSeek V3 recipe) take a different kernel from per-tensor FP8, and an unsupported shape can fall back to something slower.
- The scales. Dynamic per-token activation scales add a quantize step before every linear: read the BF16 activation, take the max, write FP8. Unfused, that is four extra kernels per layer, 320 per step for 80 layers. At about 3 us each that is 0.96 ms, 8% of the expected step: real, but it does not reach 8.3 ms alone.
- The config. Enabling FP8 often changes more than the weights. An FP8 KV cache can select a different attention backend; a new flag can disable CUDA graphs or change the captured batch sizes; a re-tuned
max_num_seqscan shrink the batch. None of these show up in a GEMM benchmark.
The measurement that separates kernel from overhead is one trace per precision of a few decode steps at a fixed batch of 64, taken with the engine's latency benchmark rather than live traffic, so the scheduler cannot change the batch between runs:
| Trace result (FP8 vs BF16) | Diagnosis | Next step |
|---|---|---|
| GEMM time up | kernel path (fallback, weight-only, bad shape) | read kernel names, check GPU arch |
| GEMM time down, many new small kernels | unfused activation quantization | fused quant, or static scales |
| GEMM and attention times down, step equal | host gaps, graphs off | compare launch gaps and graph flags |
| attention time up | FP8 KV changed the backend | pin the backend, A/B with BF16 KV |
| every kernel faster, step faster | not the GPU | look at the workload |
Kernel names settle the first row: a Marlin-style weight-only kernel, or a BF16 GEMM where an e4m3 kernel was expected, is the answer by itself.
The last row is the trap most candidates miss. If every step is faster and throughput still fell, check the output length distribution. A quantized model that drifts toward longer answers, or misses its stop token more often, emits more tokens per request, so requests per second fall while tokens per second rise. Compare output tokens per second from the engine, and the per-request token counts, before and after.
The reversal condition: the precision change is not automatically a win. In the bandwidth-bound regime FP8 weights should help on any hardware with a native path; in the compute-bound prefill regime on hardware without native FP8, BF16 is the faster choice, and the right fix is to keep BF16 for that fleet rather than tune the FP8 kernel.
What interviewers probe next
"The trace shows 320 new quantize kernels. What do you do?" Use a checkpoint with static activation scales calibrated offline, or a build that fuses the quantization into the preceding norm, then re-run the same trace to confirm the gap closed.
"Accuracy is also slightly worse. Related?" Possibly: per-tensor scales clip outlier channels. That is a quality issue, measured on an eval set, and it is separate from the speed regression unless it lengthened the outputs.
"Now it is a B200." Native FP8 and FP4 paths exist, and FP8 block scaling uses the MXFP8 recipe with power-of-two scales on Blackwell rather than Hopper's FP32 scales, so kernel selection is again the first thing to check.
How it's scored
Weak: concludes "FP8 is slower on this model" and reverts, without an expected number or a trace.
Hire: computes the expected speedup, profiles both precisions at a fixed batch, and checks kernel names for a fallback path.
Strong Hire: quantifies the gap, bounds the scale overhead with arithmetic so it can be ruled in or out, separates kernel, scale and config with one table, and checks output lengths before blaming the GPU.
Common mistakes
- Comparing live-traffic throughput, where the scheduler's batch differs between runs, instead of a fixed-batch benchmark.
- Assuming FP8 means native FP8 math on every GPU generation.
- Reading requests per second as the throughput metric when output length moved.
- Changing the KV cache dtype and the weight format in the same rollout, so neither can be blamed.