Per-tensor, per-channel, per-group, per-block: where does each quantization scale live, and what does it cost in bytes and in the kernel?

TL;DR: Count scales per weight, then ask which axis they vary along. For Llama 3.3 70B, per-channel FP32 scales cost 27 MB (0.04% of FP8 weights), 128x128 blocks cost 17 MB, but INT4 with FP16 scales per group of 128 costs 1.07 GB, 3.1% of the 34 GB of packed weights. The bigger cost is placement: scales that vary along the output or token axis are applied once in the GEMM epilogue, while scales that vary along the reduction axis must be applied inside the main loop.

How to approach it

Fix two things before any arithmetic: the format (INT8, FP8, INT4) and which tensor (weights, activations, KV). Then name the axes. A weight matrix is out x in, and the GEMM reduces over in (the K dimension), so every granularity is either constant along K or not. Ask whether activation scales are static (calibrated offline) or dynamic (computed per batch), because dynamic scales add a reduction to the forward pass.

A strong answer

The four granularities differ in how many weights share one scale and along which axis the scale changes:

The bytes, counted for Llama 3.3 70B's 68.45B linear-layer weights (seven matrices per layer, 80 layers, embeddings and LM head excluded):

per-tensor  : 7 matrices x 80 layers = 560 scales x 4 B              = 2.2 KB
per-channel : (8,192 x 3 + 1,024 x 2 + 28,672 x 2) x 80 = 6.72M x 4 B = 26.9 MB
128x128 blk : 68.45e9 / 16,384 = 4.18M scales x 4 B                   = 16.7 MB
INT4 g128   : 68.45e9 / 128 = 535M scales x 2 B (FP16)                = 1.07 GB
              against 68.45e9 x 0.5 B = 34.2 GB packed, so 3.1%, 4.125 bits/weight
NVFP4 (16)  : 68.45e9 / 16 = 4.28B E4M3 scales x 1 B                  = 4.28 GB
sanity: per-channel FP32 is 26.9 MB / 68.45 GB = 0.04% of FP8 weights, noise

Bytes stop mattering at per-channel and block granularity; they start to matter at groups of 16 to 128, which is why 4-bit formats spend design effort on the scale's own width (NVFP4 stores an 8-bit E4M3 scale per 16 values, not a 16-bit one).

The kernel cost is set by the "varies along K" column. With a per-token activation scale $s_a[m]$ and a per-channel weight scale $s_w[n]$, the scales factor out of the sum:

$$Y[m, n] = s_a[m] \, s_w[n] \sum_k A_q[m, k] \, W_q[n, k]$$

The tensor cores run the integer or FP8 product at full rate and the epilogue multiplies each output by a rank-one scale matrix, one FMA per output element. A per-group weight scale cannot be pulled out, because it changes inside the sum:

$$Y[m, n] = \sum_{g} s_w[n, g] \sum_{k \in g} A[m, k] \, W_q[n, k]$$

So the kernel must either dequantize weights to BF16 in registers before each MMA (the W4A16 path, which keeps the BF16 compute ceiling), or accumulate each group's partial sum separately and fold it into an FP32 accumulator with its scale every g elements of K. DeepSeek V3's FP8 recipe does the second at a 128-element interval, with 1x128 activation tiles and 128x128 weight blocks.

Activations add one more cost. A dynamic per-token scale needs the maximum of each token's row before quantization, a reduction over 8,192 values. Fused into the preceding RMSNorm or SiLU, it is free; run as its own kernel, it re-reads the activation.

My default for FP8 serving on Hopper: per-channel weights with per-token dynamic activations, all applied in the epilogue. Move to block scaling when the per-channel version loses accuracy on a model with activation outliers, and pay the main-loop cost knowingly. The reversal condition is precision: at 4 bits, per-channel scales are too coarse to hold accuracy on most LLMs, so groups or micro-blocks become mandatory, and the main-loop cost comes with them.

What interviewers probe next

"Why not per-input-channel scales for weights?" They vary along K, so they cannot be factored out of the dot product. They have to be folded into the activations instead, which is exactly what SmoothQuant does offline.

"Asymmetric INT4 with a zero point: what does it add?" A zero point per group, 4 bits if packed or 16 if stored as FP16, plus a correction term in the dequantization. It buys accuracy for skewed weight distributions.

"What about the KV cache?" Per-tensor KV scales fit in the attention epilogue; per-head or per-token KV scales must be read alongside each K and V tile inside the attention kernel.

How it's scored

Weak: describes the granularities as "coarse to fine" and stops there, or counts scale bytes only.

Hire: computes scale overhead for a real model, shows it is negligible for per-channel and about 3% for INT4 g128, and knows group scales are applied during dequantization.

Strong Hire: factors the dot product to show which scales can sit in the epilogue, explains the main-loop promotion that block scaling forces, prices dynamic activation scales as a fused reduction, and states when 4 bits makes fine scales compulsory.

Common mistakes

• Quoting INT4 as 4.0 bits per weight. With FP16 scales per 128 it is 4.125, and 4.25 with FP16 zero points as well.
• Putting per-group scales "in the epilogue", which computes the wrong answer as soon as a row spans more than one group.
• Computing dynamic per-token activation scales in a separate kernel, re-reading the activation the fusion was meant to avoid.
• Counting LM head and embeddings at the same granularity as the body when most recipes leave them in BF16.