← 🔢 Quantization & Numerics
Advanced
Weight-Only Quantization Versus W8A8
Weight-only quantization (W4A16, W8A16) stores small weights but multiplies in BF16, so it speeds up bandwidth-bound decode and does nothing for prefill. W8A8 quantizes activations too and runs on FP8 or INT8 tensor cores. The crossover batch is set by the GPU, about 141 on a B200, and activation outliers decide how hard W8A8 is.
Get full Premium access · ₹2,000 / $25
Every worked answer and derivation in all 12 topics, every concept page, and the full practice-test bank.
6 months · One payment · No auto-renewal
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
Quantization & NumericsAWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.→Quantization & NumericsFP8 comes in E4M3 and E5M2. Give the range and precision of each, and say which tensors use which in inference.→Quantization & NumericsPer-tensor, per-channel, per-group, per-block: where does each quantization scale live, and what does it cost in bytes and in the kernel?→KV Cache & GPU MemoryFP8 KV cache: what scales does it need, where is the error largest, and how would you validate it on long-context tasks?→Quantization & NumericsGPTQ and AWQ both produce INT4 weights with group scales. What does each one optimize, and when does one beat the other?→Quantization & NumericsWhy do activation outliers break INT8 W8A8 on LLMs, and how does SmoothQuant fix it by moving scale into the weights?→