llama cpp

Local-inference formats spend every bit where it buys the most accuracy, and datacenter formats spend them uniformly. The answer works out what a Q4_K_M file actually costs per weight, and explains why the trade that wins on one consumer GPU loses at batch 64.

One engine is built to run a model anywhere with one user; the other is built to share a datacenter GPU across hundreds. The answer turns that into memory-bandwidth arithmetic and finds the cases where the laptop wins.