Why do activation outliers break INT8 W8A8 on LLMs, and how does SmoothQuant fix it by moving scale into the weights?

TL;DR: LLM activations carry a few channels that are tens of times larger than the rest, in every token, so a per-token INT8 scale spends its 127 levels on the outliers and leaves ordinary channels with one or two. Per-channel activation scales would fix it but vary along the GEMM's reduction axis, so they cannot be factored out. SmoothQuant divides each activation channel by $s_j = \max|X_j|^{\alpha} / \max|W_j|^{1-\alpha}$ and multiplies the matching weight column by $s_j$, an exact rewrite; in a synthetic test it cuts W8A8 output error from 9.5% to 1.8%.

How to approach it

Start from where the outliers sit: along channels (the hidden dimension), consistently across tokens, not in random tokens. That one fact explains both the failure and the fix. Then ask the target format, because INT8 and FP8 respond very differently to outliers, and whether activation scales are static or dynamic per token.

A strong answer

Why INT8 fails. A W8A8 GEMM needs scales it can factor out of the dot product: per-token for activations and per-output-channel for weights, both applied in the epilogue. A per-token scale is set by that token's largest element, which is almost always in an outlier channel. Everything else in the token then shares a grid sized for the outlier:

synthetic activations, 4,096 channels, 6 outlier channels at 60x
  largest channel max / median channel max   = 73
  levels left for a typical channel          = 127 / 73 = 1.7 steps
  that is values in {-2 ... 2}, about 2.3 bits of a nominal 8

The weights are the easy side: they have no comparable outliers, and per-channel weight scales handle them.

Why the direct fix does not work. Per-input-channel activation scales $s_j$ would fix the activation grid, but $j$ is the reduction index: $\sum_j (x_j / s_j) \, w_j$ cannot be rescaled after the sum. The kernel would have to apply scales inside the main loop, losing the plain INT8 tensor-core path.

What SmoothQuant does. Because $XW^\top = (X \operatorname{diag}(s)^{-1})(W \operatorname{diag}(s))^\top$ exactly, the per-channel scale can be moved out of the runtime activations and into the weights offline. Divide activation channel $j$ by $s_j$, multiply weight input-column $j$ by $s_j$, and split the difficulty between them with one exponent $\alpha$:

$$s_j = \frac{\max_t |X_{t,j}|^{\alpha}}{\max_n |W_{n,j}|^{1-\alpha}}$$

The $1/s_j$ on activations costs nothing at runtime: it folds into the preceding RMSNorm gain for the QKV and gate/up projections, and into the rows of the producing projection for the others. The weight columns absorb $s_j$ before quantization.

import numpy as np

def q8(t, axis=None):
    """Symmetric INT8 fake-quant; axis=None per-tensor, else max over that axis."""
    amax = np.abs(t).max() if axis is None else np.abs(t).max(axis, keepdims=True)
    s = amax / 127
    return np.clip(np.round(t / s), -127, 127) * s

def w8a8_error(X, W):                       # X [tokens, in], W [out, in]
    Y = X @ W.T
    Yq = q8(X, axis=1) @ q8(W, axis=1).T    # per-token acts, per-channel weights
    return np.linalg.norm(Yq - Y) / np.linalg.norm(Y)

def smooth(X, W, alpha=0.5):
    s = np.abs(X).max(0) ** alpha / np.abs(W).max(0) ** (1 - alpha)   # [in]
    return X / s, W * s                     # X W^T is unchanged

rng = np.random.default_rng(0)
X = rng.normal(size=(512, 4096))
X[:, rng.choice(4096, 6, replace=False)] *= 60       # a few outlier channels
W = rng.normal(size=(1024, 4096)) * 0.02
Xs, Ws = smooth(X, W)
print(f"before {w8a8_error(X, W):.4f}  after {w8a8_error(Xs, Ws):.4f}")

It prints before 0.0949 after 0.0181. Sweeping $\alpha$ shows the trade directly: the outlier ratio moves from activations to weights.

alpha   act max/median   weight max/median   W8A8 relative error
0.25    25.4             2.9                 3.5%
0.50     8.6             8.6                 1.8%   (balanced, the minimum here)
0.75     2.9            25.4                 3.5%
1.00     1.0            74.8                 9.8%   (all difficulty pushed to weights)
synthetic data; the SmoothQuant authors use 0.5 for most models and a larger
alpha where activation outliers are more severe (arXiv 2211.10438)

The reversal condition is the format. FP8 E4M3 is floating point, so a small value keeps about 3 mantissa bits of relative precision however large the token's maximum is, until it falls below the subnormal range. On the same synthetic data with per-token FP8 activations, the error is 3.3% before smoothing and 2.9% after. So on Hopper and Blackwell, where FP8 W8A8 runs at twice the BF16 tensor-core rate, SmoothQuant is a minor refinement. It earns its keep on INT8 paths: Ampere's A100 (INT8 tensor cores, no FP8), many edge NPUs, and per-tensor static activation scales, where the same data shows 21% error dropping to 2.9%.

What interviewers probe next

"Why not keep outlier channels in FP16 like LLM.int8()?" That splits the GEMM into a dense INT8 part and a sparse FP16 part with a gather, which costs throughput; SmoothQuant keeps one dense INT8 GEMM.

"Which layers can be smoothed for free?" Any linear whose input passes through an op that can absorb a per-channel factor offline: a norm gain, or a preceding linear's output rows. A residual-stream input with no such op needs an explicit scale multiply.

"Now the target is W4A4." Smoothing alone is not enough at 4 bits. Rotation methods that spread outliers across all channels, or block-scaled FP4 with a scale per 16 values, are the tools there.

How it's scored

Weak: says outliers "hurt accuracy" and proposes clipping them, or proposes per-channel activation scales without seeing they block the INT8 GEMM.

Hire: explains the channel-wise structure of outliers, why per-token scales waste the grid, and the exact rewrite that moves scale into the weights.

Strong Hire: quantifies levels lost to the outlier, explains the alpha trade with numbers, folds $1/s$ into the preceding op, and states that FP8's floating-point grid makes smoothing marginal on Hopper.

Common mistakes

• Clipping outlier activations. Those channels carry real signal, and clipping them damages the model more than coarse rounding.
• Applying the smoothing factor at runtime as a separate multiply, adding a pass over the activations.
• Choosing alpha 1.0 because it eliminates activation outliers, then discovering the weights now carry them.
• Computing smoothing factors from a calibration set too small to contain the outlier channels' true maxima.