What Q4_K_M means: reading GGUF quantization names

Open any GGUF repository and you'll see a dozen files that differ only in a short code at the end: Q4_K_M, Q5_K_S, Q6_K, Q8_0, IQ4_XS. Each code is a recipe for how the model's weights were squeezed. Once you can read it, you can guess the file size and roughly how much quality you're trading before you download 20 GB.

The number and the letters

Q means quantized. The number after it is the base bit width for most of the weights, so Q4 stores them at about four bits and Q8 at about eight. The K means the file uses llama.cpp's k-quants, which group weights into blocks and super-blocks that each carry their own scale. The Hugging Face GGUF reference lists every type with its layout. Its Q4_K entry works out to 4.5 bits per weight and Q6_K to 6.5625, because the scales take up space too.

What S, M and L change

The trailing letter is small, medium or large. The original k-quants pull request spells out the recipes. Q4_K_S uses the 4-bit type for every tensor. Q4_K_M gives half of two sensitive tensor groups, the attention value weights and one feed-forward projection, the 6-bit type and keeps the rest at 4 bits. So M and L spend a little more space where errors hurt the most. llama.cpp has tuned these recipes since 2023, so treat the PR as the idea and your file's metadata as the final word.

File size is close to parameters times bits per weight, divided by eight. That's a handy check before you download anything.

Why Q4_K_M is bigger than 4 bits

The llama.cpp team publishes measured sizes for Llama 3.1 8B in the quantize tool's README. They show how far the real files sit from the name:

Q4_K_M comes out near 4.9 bits because of the block scales and the tensors it bumps up to 6 bits. The same README sums up why anyone bothers:

Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference. From the llama.cpp quantize README

Estimate a download before you grab it

Say you're looking at a 32B model at Q4_K_M. Multiply 32 billion by 4.9 bits and divide by eight: about 19.6 GB, or roughly 18 GiB. That's an arithmetic estimate. Different architectures keep different tensors at higher precision, so check the actual file size on the repository page, which is usually listed next to each download.

Two other families show up in most repositories, and they follow different rules from the K types.

IQ types and the importance matrix

IQ4_XS, IQ3_S and friends use an importance matrix: a file built by running sample text through the model and recording which weights matter most. The quantizer then spends its bits where they count. The Hugging Face table puts IQ4_XS at 4.25 bits per weight, a little under Q4_K. The quantize README says the accuracy loss from quantizing "can be minimized by using a suitable imatrix file", which is why the smallest usable files at 2 and 3 bits are nearly always IQ types.

The catch is that the quality of an IQ file depends on the sample text used to build its matrix. Two IQ4_XS uploads of the same model from different people can behave differently, so note who made the file.

Q8_0 and Q4_0

The _0 and _1 types are the older, simpler formats: one scale per block of 32 weights and nothing else. Hugging Face marks them as legacy. Q8_0 is still everywhere, though, because at eight bits there's so little loss that the simple method is good enough. In the Llama 3.1 8B table it's 8.5 bits per weight and 7.95 GiB. Q4_0 is a different story. At four bits the K and IQ types do better for the same size, so you rarely need it unless your runtime or hardware only supports the old types.

Knowing the names makes the choice shorter. You still have to try the file on your own work.

Start from memory

Work out what you can afford first. The VRAM calculator takes your GPU and a model and shows the weights, the context and the runtime overhead as separate lines. Pick the largest quant that leaves room for the context you really use. A Q5_K_M that fits with 4K of context is a worse choice than a Q4_K_M that fits with the 16K your documents need. Our VRAM requirements guide covers where the extra memory goes.

Then test quality on your own work

Bit width tells you how much was thrown away. It can't tell you whether your tasks notice. Run the same prompts on two neighbouring quants, say Q4_K_M and Q5_K_M, and compare the answers you'd accept. If they tie, keep the smaller one and enjoy the headroom. The quantization guide explains the trade-offs in more depth, and the GPU reports show which quants people run on cards like yours. When you settle on one, share your setup with the exact file name, so the next person with your card doesn't have to guess.