07You quantized a model and perplexity barely moved. How do you evaluate it before it serves traffic, and why is perplexity not enough?▼mediumNewOpenAIAnthropicMeta○ sign inPerplexity averages over the tokens a model finds easy and scores it on text it did not write. The answer builds the evaluation that catches what quantization actually breaks, and shows why a one-point drop on a benchmark can be noise or a real regression.Open full answer →
08Which layers should stay in higher precision after quantization, and how do you find them with a sensitivity sweep?▼hardNewNVIDIAMetaQualcomm○ sign inA few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.Open full answer →