01AWQ INT4 beat FP8 at batch 1, lost to it at batch 256, and cost 4 points on a coding eval. Explain both and pick a format.▼hard★ EssentialNewNVIDIAOpenAIMetaunlockedWeight-only INT4 shrinks the bytes but keeps the arithmetic in BF16. The answer finds the batch size where that bargain flips, which does not depend on the model, then explains why a code benchmark notices before perplexity does.Open full answer →
04GPTQ and AWQ both produce INT4 weights with group scales. What does each one optimize, and when does one beat the other?▼mediumNewNVIDIARed HatMetaunlockedThe two methods ship identical file formats and run on identical kernels, so the difference is entirely in how the rounding was chosen. The answer shows the one design choice that makes one of them fragile when traffic stops looking like the calibration set.Open full answer →