How to compare two quantizations on your own tasks

What quantization changes

A quantized file stores each weight in fewer bits, so it needs less memory and usually runs faster. Hugging Face's overview describes the trade:

"Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible." (Hugging Face: quantization overview)

The size difference is big. llama.cpp's own table for Llama 3.1 8B lists the Q4_K_M file at 4.58 GiB, Q8_0 at 7.95 GiB and the 16-bit original at 14.96 GiB (llama.cpp quantize README). Dropping from Q8_0 to Q4_K_M saves over 3 GB on a model this small, and far more on a big one, so it's worth knowing what you give up for it.

Change one thing only

Take both files from the same model release, from the same uploader if you can, and write down where each came from and its exact label. If they differ in anything besides precision, note that, because your test is now measuring it too.

Run both on the same engine version, hardware, context size and sampling settings. Change the context at the same time and you may still land on a setup you like, but you won't know whether the quantization had anything to do with it. If names like Q5_K_S and IQ4_XS mean nothing to you yet, the quantization guide decodes them.

Checks you can score quickly

Pick tasks you care about and decide what counts as a pass for each one before you run anything. For code, that could be the existing tests passing with the function signature unchanged. For extraction, compare the fields with an answer you already know, down to the exact format you asked for.

Five to ten tasks will do to start. Include the kinds of task where small models tend to slip for you, such as long instructions or exact formats, since a lower-bit file usually shows its losses there first.

Grade without knowing which is which

Strip the labels before you review if you can. An answer you know came from the bigger file looks better to you than it should. Rename the outputs A and B, score them, and only then check which was which. Keep the failed outputs with a note on why they failed. When you revisit the choice later, they'll tell you more than the passes do.

If the smaller file passes all eight of your checks, use it for this work. Once you start pointing it at something new, add a task for that to the set and rerun it.

Time the whole task

Record memory use if your runtime shows it, along with the wait before the first word, the output length and the total time. If you note tokens per second as well, write down how you measured it. The Ollama speed guide shows one way to get a clean figure.

Watch answer length

At the same tokens per second, a shorter answer finishes sooner, and a faster file that rambles can still take longer overall. Check the task result before you call either timing an improvement.

Run each prompt more than once and keep all the results. When outputs vary between runs, compare the pattern across runs and ignore the single best one. Warmup and repeated prompts should be the same for both files, and noted. The comparison guide covers the settings that change speed without you noticing.

A table beats a rule of thumb

Fill in a small table for the two files. Having the trade written down keeps you from remembering only the half you liked.

Pick per job if you need to

If the smaller file passes your checks and the memory or time it saves matters to you, use it. If its failures cost you more than that memory is worth, use the larger one. Keeping both is fine when different jobs want different trade-offs.

Before you download anything, the calculator can tell you which quantizations fit. After that, your own task results make the call. Anyone comparing the same two files will need your pass checks and both file names to make sense of your result, so include them if you share it.