08Which layers should stay in higher precision after quantization, and how do you find them with a sensitivity sweep?▼hardNewNVIDIAMetaQualcomm○ sign inA few matrices cause most of the damage when a model is quantized, and keeping them wide costs little memory. The answer builds the sweep that finds them, prices it in GPU hours, and explains why the sum of single-layer damages is not the damage of the mix.Open full answer →
10GGUF k-quants and EXL2 mix bit-widths inside one model. What do they trade, and where do they fit next to datacenter GPU formats?▼mediumNewMetaQualcommMistral AI○ sign inLocal-inference formats spend every bit where it buys the most accuracy, and datacenter formats spend them uniformly. The answer works out what a Q4_K_M file actually costs per weight, and explains why the trade that wins on one consumer GPU loses at batch 64.Open full answer →