llamaperf
← All articles

Quantization

What Q4_K_M means: reading GGUF quantization names

By llamaperf · · 6 min read

Quick answer

The number is roughly how many bits each weight gets. K means the file uses llama.cpp's k-quant method, and the last letter (S, M or L) says how many of the most sensitive tensors get a bigger type. In practice Q4_K_M lands near 4.9 bits per weight, so an 8B model is around 4.6 GiB before you add any context.

How to read the name

Open any GGUF repository and you'll find a dozen files that differ only in a short code at the end: Q4_K_M, Q5_K_S, Q6_K, Q8_0, IQ4_XS. Each code describes how the file stores quantized weights. If you can read it, you can guess the file size, and roughly how much quality you're giving up, before you download 20 GB.

The number and the letters

Q means quantized. The number after it is the base bit width for most of the weights, about four bits for Q4 and about eight for Q8. K says the file uses llama.cpp's k-quants, which group weights into blocks and super-blocks that each carry their own scale, and those scales take up room as well. The Hugging Face GGUF reference lists every type with its layout: its Q4_K entry works out to 4.5 bits per weight, and Q6_K to 6.5625.

What S, M and L change

The trailing letter stands for small, medium or large, and the original k-quants pull request spells out each recipe. Q4_K_S uses the 4-bit type for every tensor. Q4_K_M takes half of two sensitive tensor groups, the attention value weights and one feed-forward projection, up to the 6-bit type and leaves everything else at 4 bits, so it spends a little extra space where rounding errors do the most damage. llama.cpp has kept tuning these recipes since 2023. Read the PR for the idea and your file's metadata for what was done to it.

From bits per weight to file size

Multiply the parameter count by the bits per weight, divide by eight, and you have the file size in bytes, give or take a little.

Why Q4_K_M is bigger than 4 bits

The quantize tool's README has measured sizes for Llama 3.1 8B, and they show how far a real file sits from its name:

TypeBits per weightSize (GiB)
Q4_K_S4.66724.36
Q4_K_M4.89444.58
Q5_K_M5.70365.33
Q6_K6.56336.14
Q8_08.50087.95
F1616.000514.96

Block scales plus the tensors bumped up to 6 bits are what take Q4_K_M to nearly 4.9 bits. The same README gives the reason anyone quantizes in the first place:

Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference. From the llama.cpp quantize README

Estimate a download before you grab it

Say you're eyeing a 32B model at Q4_K_M. 32 billion times 4.9 bits, divided by eight, is about 19.6 GB, or roughly 18 GiB. Architectures differ in which tensors they keep at higher precision, so the real file can land a bit either side of that sum. Most repository pages list the size next to each download, and that figure beats your arithmetic.

IQ quants and the older _0 types

IQ types and the importance matrix

Next to the K files you'll usually see IQ4_XS, IQ3_S and other IQ types. These use an importance matrix, a file made by running sample text through the model and recording which weights matter most, and the quantizer spends its bits on those. The Hugging Face table puts IQ4_XS at 4.25 bits per weight, a little under Q4_K. According to the quantize README, the accuracy loss from quantizing "can be minimized by using a suitable imatrix file", which is why the smallest usable files, at 2 and 3 bits, are nearly always IQ types.

An IQ file is only as good as the sample text behind its matrix, though. Two IQ4_XS uploads of the same model by different people can behave differently, so write down who made yours.

Q8_0 and Q4_0

The _0 and _1 types are the older, simpler formats, with one scale per block of 32 weights and nothing more, and Hugging Face marks them as legacy. Q8_0 remains common because at eight bits the simple method loses little precision, and in the Llama 3.1 8B table it comes to 8.5 bits per weight and 7.95 GiB. At four bits, K and IQ types preserve quality better than Q4_0 at the same size, and the only good reason to pick it is a runtime or chip that supports nothing newer.

Picking a quant for your card

Start from memory

Work out what you can afford before you think about quality. The VRAM calculator takes your GPU and a model and lists the weights, the context and the runtime overhead on separate lines. Pick the largest quant that still leaves room for the context you really use: if your documents need 16K, a Q4_K_M that fits with 16K is a better buy than a Q5_K_M that only fits with 4K. Our VRAM requirements guide covers where the rest of the memory goes.

Then test quality on your own work

Bit width describes precision. Test whether the reduction affects your tasks. Run the same prompts on two neighbouring quants, Q4_K_M and Q5_K_M for example, and count the answers you'd accept from each. If they tie, keep the smaller file and give the spare memory to context. The quantization guide goes further into the trade-offs, and the GPU reports show which quants people run on cards like yours. Once you've picked one, put the full file name in your setup report, uploader included, since two files with the same quant name can come from different recipes and different importance matrices.

Frequently asked questions

What does Q4_K_M mean?

It has three parts. Q4 is the base width, about four bits for most weights. K is llama.cpp's k-quant method, which stores a scale for each block of weights. M is the medium recipe, which keeps a few sensitive tensors at 6 bits. Together they average about 4.9 bits per weight on Llama 3.1 8B.

What's the difference between Q4_K_S and Q4_K_M?

Q4_K_S keeps every tensor at the 4-bit type, while Q4_K_M moves part of the attention value and feed-forward weights up to 6 bits. On Llama 3.1 8B the gap is about 0.2 GiB. Unless those 200 MB decide whether the model fits, most people take M.

Is a smaller model at Q8_0 smarter than a bigger one at Q4_K_M?

Nobody can answer that in general. A bigger model at 4 bits often knows more, and a smaller one at 8 bits loses almost nothing to quantization, so it comes down to your tasks. Try both and keep whichever passes more of them at a speed you can live with.

How much VRAM does a Q4_K_M model need?

Add the context cache and the runtime's own buffers to the file size. A 4.6 GiB file gets uncomfortable on a 6 GB card once you give it a long context. The llamaperf calculator adds all of it up for your card and context length.

What are IQ quants?

They're the types whose names start with IQ, like IQ4_XS and IQ3_S. The quantizer uses an importance matrix, built from sample text, to decide which weights keep more precision. At very small sizes they beat K types on quality, though how well a given file does depends on the sample text its uploader used.

Why is my Q4_K_M file bigger than 4 bits per weight suggests?

Each block of weights stores its scales too, and the M recipe keeps some tensors at 6 bits. Between them they push Q4_K_M to about 4.9 bits per weight. Some models also store the embeddings or the output layer at higher precision, which adds a bit more.