07You quantized a model and perplexity barely moved. How do you evaluate it before it serves traffic, and why is perplexity not enough?▼mediumNewOpenAIAnthropicMeta○ sign inPerplexity averages over the tokens a model finds easy and scores it on text it did not write. The answer builds the evaluation that catches what quantization actually breaks, and shows why a one-point drop on a benchmark can be noise or a real regression.Open full answer →
11Design the rollout of a new model version so that neither quality nor latency regresses in production. How do you know when to promote?▼mediumNewOpenAIAnthropicGoogle DeepMind◆ premiumA canary is only as good as the number of requests behind each verdict. The answer sizes the canary from the statistics of the regression it must catch, then stages the rollout around that number.Open full answer →
08A quantized model passes its perplexity checks but fails long-context retrieval. Debug it.▼hardNewAnthropicGoogle DeepMindCohere○ sign inPerplexity matches the BF16 baseline to two decimals and the needle-in-a-haystack score collapses past 16K. The answer explains why the two checks measure different things, isolates weights, KV and config with a small grid, and shows a measured case of a quantization scheme that only breaks at length.Open full answer →