Quantization chains: bits with scales, where INT4 loses, and what rounds to zero

Numerics questions reward exact arithmetic about formats. Each rounding example below was checked against the real format with ml_dtypes, so the zeros and the 256 follow from the stated rounding behavior. These CPU checks do not execute a GPU kernel.

Bits per weight, scales included

INT4 g128   4 bits + one FP16 scale per 128 weights = 4 + 16/128 = 4.125 bits
MXFP4       4 bits + one E8M0 scale per 32          = 4 + 8/32  = 4.25 bits
NVFP4       4 bits + one E4M3 scale per 16          = 4 + 8/16  = 4.5 bits,
            plus one FP32 scale per tensor (negligible)
FP8 per-channel  Llama 3.3 70B linear layers: 26.9 MB / 68.45 GB = 0.039% extra
sanity   finer blocks cost bits; NVFP4 spends 9% more than INT4 g128 on scales

Quoting "4-bit" for all three hides a 9% difference in bytes, and bytes are what decode reads.

Drilled in: Scale granularity and what it costs · NVFP4 versus MXFP4

70B weight bytes by format

inputs   Llama 3.3 70B: 70.55e9 parameters; embeddings and LM head
         2 x 128,256 x 8,192 = 2.10e9 kept in BF16; body 68.45e9
BF16     70.55e9 x 2 B = 141.1 GB
FP8      68.45e9 x 1 B + 2.10e9 x 2 B = 72.7 GB
INT4 g128  68.45e9 x 4.125 / 8 B + 2.10e9 x 2 B = 39.5 GB
sanity   INT4 is 3.6x smaller than BF16, not 4x: the scales and the BF16
         edges take the rest

Most recipes leave the embedding and LM head unquantized. Forgetting them understates every quantized footprint by about 2 GB on this model.

Drilled in: Scale granularity and what it costs

Where INT4 W4A16 loses to FP8

inputs   Llama 3.3 70B at TP=2 on H100 SXM; BF16 peak 989, FP8 peak 1,979 TFLOP/s
         every matrix quantized, as in the linked question (36.4 GB INT4)
         W4A16 multiplies on BF16 tensor cores; FP8 W8A8 on FP8 tensor cores
batch 1  INT4 reads 36.4 GB / 2 / 3.35 TB/s = 5.4 ms; FP8 reads 10.5 ms
compute  INT4 step = 2 x 70.55e9 x B / (2 x 989e12) = 71 us x B
cross    INT4 compute = FP8 read time when B = 989 / (2 x 3.35) = 148
sanity   INT4 becomes compute-bound near 76, but its floor stays below FP8
         until about 148. This weights-only model omits conversion, KV and collectives.

INT4 weight-only quantization shrinks reads and leaves the math in BF16. Native W8A8 or FP4 raises the arithmetic ceiling on supporting hardware. Compare matched workloads and quality gates before choosing a format.

Drilled in: INT4 wins at batch 1, FP8 at batch 256 · Weight-only versus W8A8

E4M3 against E5M2

E4M3     max 448, smallest normal 2^-6, smallest subnormal 2^-9,
         3 mantissa bits: unit roundoff 2^-4 = 6.25%
E5M2     max 57,344, 2 mantissa bits: unit roundoff 2^-3 = 12.5%
BF16     FP32's range, 7 mantissa bits: unit roundoff 2^-8 = 0.39%
sanity   E4M3 rounds twice as finely as E5M2; a scale can fix range,
         nothing restores mantissa bits

Inference uses E4M3 with scales for weights, activations and KV. E5M2 is for tensors whose range cannot be scaled ahead of time.

Drilled in: FP8 E4M3 versus E5M2 · Number formats for inference

One outlier sets the scale

inputs   an E4M3 tensor whose largest magnitude is 100, so the per-tensor
         scale is 100 / 448 = 0.223
0.0002   0.0002 / 0.223 = 0.0009, below half the smallest subnormal
         (0.00098): stored as 0, a 100% error
0.0004   stored as 0.000436, a 9% error
per block  a block whose largest magnitude is 1.0 gets scale 1 / 448;
         both values land in the normal range with a 4.1% error
sanity   one outlier spends the format's range on itself and pushes small
         values into the subnormals or to zero

This is the case for per-channel and per-block scales, and for moving activation outliers into the weights before quantizing.

Drilled in: SmoothQuant and activation outliers

MXFP4 against NVFP4 scales

inputs   an FP4 (E2M1) block whose largest magnitude is 5.0;
         E2M1 represents 0, 0.5, 1, 1.5, 2, 3, 4, 6
MXFP4    power-of-two scale 1: 5.0 rounds to 4.0, a 20% error on the
         block's largest value
NVFP4    E4M3 scale nearest 5 / 6 is 0.8125: 5.0 / 0.8125 = 6.15 rounds to 6,
         stored 6 x 0.8125 = 4.875, a 2.5% error
sanity   the fractional scale, more than the smaller block, is what lets
         NVFP4 map a block's maximum onto 6

A power-of-two scale can leave a block's largest value anywhere in [4, 8), so the error depends on where the maximum happens to fall.

Drilled in: NVFP4 versus MXFP4 · Block-scaled FP4

A BF16 sum that stops at 256

inputs   add 1.0 to a BF16 running total 1,000 times
BF16     8 significant bits: from 256 to 512 the spacing is 2
step     256 + 1 = 257 is a tie between 256 and 258; round-to-even keeps 256
result   the total ends at 256, not 1,000
sanity   accumulate in FP32, whose spacing at 256 is 2^-15

An error that grows with sequence length is the signature of state accumulated in a 16-bit type: short inputs pass and long ones drift.

Drilled in: BF16 accumulation bug at long context

FP16 overflow in a norm

inputs   RMSNorm over 4,096 activations, each 4.0, summed in FP16
sum      4,096 x 4.0^2 = 65,536
FP16     largest finite value 65,504: the sum is inf, and the output NaN
sanity   BF16 would not overflow here, but it would round the sum;
         norms accumulate in FP32 for both reasons

FP16 buys its finer rounding (10 stored mantissa bits to BF16's 7) with a largest value of 65,504. Fused norm kernels widen to FP32 before they reduce for that reason.

Drilled in: Fused RMSNorm in Triton