Number Formats for Inference: BF16, FP8, INT4, FP4

TL;DR: A format is three decisions: bytes per element (decode time and memory scale with it), the split between exponent and mantissa bits (range versus precision), and whether the tensor cores multiply it natively. Llama 3.3 70B weighs 141.1 GB in BF16, 70.6 GB in FP8 and 39.7 GB in NVFP4, and only the last two have tensor-core paths that also double or quadruple the BF16 compute peak (on Hopper for FP8, on Blackwell for FP4).

Storage format versus compute format

A tensor has a format it is stored in and a format it is multiplied in, and they need not match. Weight-only INT4 stores 4-bit integers and multiplies in BF16 after dequantizing in registers. FP8 W8A8 stores and multiplies in FP8. The storage format sets bytes moved, which is the whole cost of bandwidth-bound decode. The compute format sets the tensor-core ceiling, which is the cost of compute-bound prefill and large-batch decode. Mixing them up is the most common reason a quantization plan disappoints.

Every low-precision format also carries scales. A 4-bit or 8-bit grid only covers real values after multiplying by a scale chosen per tensor, per channel, per group of 64 to 128 weights, or per block of 16 to 32 elements. Finer granularity fits outliers better and costs scale bytes.

The formats

Floating-point formats spend bits on exponent (range) or fraction bits (precision). In a normal binade beginning at $2^e$, spacing is $2^{e-m}$ for $m$ fraction bits. Relative to the binade's lower endpoint, that step is $2^{-m}$: 0.78125% for BF16 and 12.5% for E4M3. Relative to an arbitrary value within the binade, the step is smaller. These are spacing figures, not per-element errors.

Round-to-nearest has the usual normal-range error bound $u=2^{-(m+1)}$: 0.390625% for BF16 and 6.25% for E4M3. Underflow and clipping need separate analysis. E4M3 rounds $2^{-11}$ to zero, a 100% relative error; a unit scale clips 600 to 448 under saturating conversion. The FP8 format paper, arXiv v1 defines the E4M3 encoding used here.

TFLOP/s are NVIDIA's dense figures (the datasheets list sparse peaks at twice these). BF16 has FP32's exponent width but fewer fraction bits, so its largest finite value is slightly smaller. FP32 accumulation also does not prevent overflow in a later narrow cast. E4M3 trades range for precision and is the inference choice for weights and activations; E5M2 has the range that gradients need. FP4 E2M1 is so coarse that it is only usable with block scales: its eight magnitudes are listed by the snippet below.

def fp_values(e_bits, m_bits, bias, fn=False):
    """All finite non-negative values of a tiny float format (no sign bit)."""
    vals = []
    for e in range(2 ** e_bits):
        for m in range(2 ** m_bits):
            if fn and e == 2 ** e_bits - 1 and m == 2 ** m_bits - 1:
                continue                  # E4M3 "fn": only S.1111.111 is NaN
            if not fn and e_bits > 2 and e == 2 ** e_bits - 1:
                continue                  # IEEE-style: top exponent is inf/NaN
            if e == 0:
                vals.append(m / 2 ** m_bits * 2.0 ** (1 - bias))       # subnormal
            else:
                vals.append((1 + m / 2 ** m_bits) * 2.0 ** (e - bias))
    return sorted(set(vals))

print(fp_values(2, 1, bias=1))            # E2M1: [0, 0.5, 1, 1.5, 2, 3, 4, 6]
print(fp_values(4, 3, bias=7, fn=True)[-1])   # E4M3 max: 448
print(fp_values(5, 2, bias=15)[-1])           # E5M2 max: 57344

Worked example: one model, four formats

model    Llama 3.3 70B, 70,553,706,496 parameters (computed from config.json)
bytes    BF16 2, FP8 1, INT4 group-128 0.516 (4 bits + 16-bit scale per 128),
         NVFP4 0.5625 (4 bits + one 8-bit scale per 16)

BF16     70.55e9 x 2      = 141.1 GB (131.4 GiB)  > one H100's 80 GB
FP8      70.55e9 x 1      =  70.6 GB  (65.7 GiB)  one H100, about 9 GB left
INT4 g128 70.55e9 x 0.516 =  36.4 GB  (33.9 GiB)
NVFP4    70.55e9 x 0.5625 =  39.7 GB  (37.0 GiB)

batch-1 decode floor on one B200 (8 TB/s), weights only:
BF16  141.1 GB / 8e12 = 17.6 ms    FP8  8.8 ms    NVFP4  5.0 ms
sanity: time ratio 17.6 / 5.0 = 3.6, the byte ratio 2 / 0.5625; at batch 1
        the format is a pure bandwidth lever

Choosing, by tensor

Where it breaks

Accumulation is the quiet requirement. Products of FP8 or FP4 operands are summed in FP32 or at least wider internal precision; a kernel that accumulates a long dot product in a narrow format loses the low bits of every partial sum. DeepSeek V3's report describes promoting FP8 partial sums to FP32 on CUDA cores at fixed intervals along K for this reason. E4M3 saturates at 448, so a per-tensor scale set from a calibration batch clips any later activation outlier. INT4 and FP4 at the same 4 bits behave differently: an integer grid is uniform, FP4's eight magnitudes crowd near zero, which suits bell-shaped weights inside a block but needs a scale per 16 or 32 elements to work at all.

What interviewers listen for

They want bytes, range and native-math support kept separate. "INT4 is faster than FP8" without saying weight-only, which batch, and which GPU is a weak answer. Strong candidates quote dense peaks, compute weight sizes in GB with the scale overhead included, and say that E4M3's 448 maximum is why scales exist.

Say it in 30 seconds

Pick a format by three properties: bytes per element, which sets decode speed and memory; exponent versus mantissa bits, which sets range and precision; and whether the tensor cores multiply it. BF16 is the baseline, FP8 E4M3 halves the bytes and doubles the dense peak on Hopper, and 4-bit formats halve the bytes again, with native FP4 math only on Blackwell. Llama 70B goes from 141 GB in BF16 to 71 GB in FP8 and about 40 GB in NVFP4.