TL;DR: INT4 weight-only (W4A16) cuts weight bytes about 3.9x but still multiplies on the BF16 tensor cores, while FP8 W8A8 halves the bytes and runs on FP8 tensor cores at twice the BF16 peak. On an H100 the INT4 step turns compute-bound at batch , for any dense model, and past that FP8 is faster: 10.5 ms against 18.3 ms per step for a 70B at TP=2 and batch 256, before INT4's dequantization overhead. The 4-point coding drop is a calibration problem first: AWQ chooses which channels to protect from activation statistics, and prose calibration data protects the wrong ones for code.
How to approach it
Split the question in two, because they have unrelated causes: the speed reversal is roofline arithmetic, the accuracy drop is calibration. For speed, ask the batch sizes the service actually runs at and the GPU generation, since Blackwell changes the answer. For accuracy, ask what the calibration set was, the group size, which layers were left unquantized, and whether the eval measures code execution or a proxy.
A strong answer
Why INT4 wins small and loses large. Model one decode step for a dense model with parameters by the roofline: time is the larger of bytes over bandwidth and FLOPs over the peak of the tensor cores the kernel actually uses. INT4 weight-only kernels dequantize to BF16 in registers and multiply in BF16, so they keep the BF16 compute ceiling. FP8 W8A8 quantizes activations too and multiplies in FP8:
| Format (H100 SXM) | Bytes per weight | Compute ceiling, dense | Memory-bound below batch |
|---|---|---|---|
| BF16 | 2 | 989 TFLOP/s | 295 |
| FP8 W8A8 | 1 | 1,979 TFLOP/s | 295 |
| INT4 W4A16, group 128 | 0.516 (4 bits + a 16-bit scale per 128) | 989 TFLOP/s | about 76 |
The crossover against FP8 falls out of one equation, INT4's compute time equal to FP8's memory time:
cancels, so the crossover is a property of the hardware, not of the model. For Llama 3.3 70B at TP=2, weight time per step only:
batch BF16 FP8 W8A8 INT4 W4A16 g128
1 21.1 ms 10.5 ms 5.4 ms (INT4 memory-bound, 1.9x faster than FP8)
128 21.1 ms 10.5 ms 9.1 ms (INT4 now compute-bound)
148 21.1 ms 10.5 ms 10.6 ms crossover
256 21.1 ms 10.5 ms 18.3 ms (FP8 1.7x faster)
sanity: at batch 512 INT4 and BF16 meet at 36.5 ms, both on the BF16 ceiling
That is the ideal. Real INT4 kernels also spend instructions unpacking nibbles and applying scales, which lowers their effective compute throughput, so the measured crossover arrives earlier than 148. Marlin-class kernels exist precisely to keep dequantization hidden under memory loads at small batch; a fallback kernel without that design can be slower than BF16 from batch 16 up. Before trusting any batch-256 number, confirm which kernel the engine selected.
Why the coding eval dropped. AWQ protects the roughly 1% of weight channels whose inputs carry the largest activations, measured on a calibration set, by scaling them before rounding. If that set is web prose, the activation statistics of code tokens (indentation, identifiers, rare symbols) are underrepresented, and the channels that matter for code are rounded hard. Perplexity on general text barely moves, because it is the distribution the calibration saw. The remaining suspects, in order: quantizing the LM head or embeddings, group size (64 instead of 128 costs 0.125 more bits per weight and buys accuracy), and a few outlier layers that should stay in higher precision.
The decision for a fleet that serves both small-batch interactive and large-batch traffic on H100s: FP8 W8A8 as the default, because it is near the BF16 baseline in quality on most evaluations and wins at the batch sizes where throughput is earned. Use INT4 where batch stays small, such as single-user, edge or latency-critical tiers, and recalibrate it with in-domain data, including code.
The reversal condition is the hardware. On Blackwell, NVFP4 is a 4-bit format that multiplies on FP4 tensor cores (9,000 dense TFLOP/s on a B200, four times its BF16 peak), so the bytes and the arithmetic shrink together and the crossover argument against 4-bit disappears. What remains is the accuracy question.
What interviewers probe next
"Why not W4A4 INT4 on Hopper too?" Hopper has no INT4 tensor-core path, and 4-bit activations without block scaling lose too much on outlier channels. Block-scaled FP4 with per-16-element scales is what made 4-bit activations practical.
"What about the KV cache?" Weight format and KV format are independent. At long context KV reads can exceed weight reads, so an FP8 KV cache may matter more than any weight format.
"How do you evaluate the quantized model?" On the tasks that will be served, scored by execution (pass@1 on held-out code, exact match on structured outputs) with long-context items included, against the same prompts in BF16. Perplexity is a smoke test.
How it's scored
Weak: says "INT4 is smaller so it is always faster", or attributes the accuracy drop to "quantization noise" with no mechanism.
Hire: explains that weight-only INT4 still computes in BF16, locates the batch where it turns compute-bound, and names calibration data and group size as accuracy levers.
Strong Hire: derives the model-independent crossover near 148, accounts for dequantization overhead and kernel selection, ties AWQ's salient-channel selection to code activations, and explains why NVFP4 on Blackwell changes the decision.
Common mistakes
- Comparing formats at batch 1 only, then deploying the winner to a fleet that runs at batch 200.
- Quoting the H100's FP8 peak as 3,958 TFLOP/s. That is the sparse figure; dense is 1,979.
- Forgetting the group scales: INT4 at group 128 is 4.125 bits per weight, not 4.
- Recalibrating with more of the same prose and expecting the code score to recover.