01AWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.▼hard★ EssentialNewNVIDIAOpenAIMetaunlockedWeight-only INT4 shrinks the bytes but keeps the arithmetic in BF16. The answer finds the batch size where that bargain flips, which does not depend on the model, then explains why a code benchmark notices before perplexity does.Open full answer →
04GPTQ and AWQ both produce INT4 weights with group scales. What does each one optimize, and when does one beat the other?▼mediumNewNVIDIARed HatMetaunlockedThe two methods ship identical file formats and run on identical kernels, so the difference is entirely in how the rounding was chosen. The answer shows the one design choice that makes one of them fragile when traffic stops looking like the calibration set.Open full answer →
08Which layers should stay in higher precision after quantization, and how do you find them with a sensitivity sweep?▼hardNewNVIDIAMetaQualcomm○ sign inA few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.Open full answer →
14A quantized MoE holds its benchmark scores but loses accuracy on rare domains, traced to rarely routed experts. Explain why and fix it.▼expertNewMistral AITogether AIFireworks AI◆ premiumCalibration data reaches each expert only through the router, so the experts that matter for rare domains see almost none of it. The answer counts the tokens, shows the point where a Hessian-based quantizer becomes worse than plain rounding, and builds a recipe that does not depend on luck.Open full answer →