Quantization for local LLMs

A quantized LLM stores its weights in fewer bits than the original model — usually 4, 5, or 8 bits per weight instead of 16. This makes the model 2–4× smaller (and faster), at the cost of some quality. Modern 4-bit quants (Q4_K_M and friends) preserve roughly 95–98% of the original model's quality, which is invisible in normal use. If your hardware can fit a Q4_K_M version of a model, that's almost always the right starting point.

Open-weight LLMs are published in 16-bit precision (BF16 or FP16), which means each parameter takes 2 bytes. A 70B model is therefore about 140GB of weights — far more than fits in any consumer GPU and most workstation cards. Without quantization, running these models locally would be impossible for almost everyone.

Quantization compresses the weights into a smaller representation. The compression isn't lossless, but for inference it turns out you can throw away a lot of precision before the model's outputs change in any noticeable way. A 4-bit quant uses one quarter of the memory of the original, which means a 70B model that needed an A100 (80GB) suddenly fits on a single 48GB card or a Mac with 64GB of unified memory.

The table below shows the major GGUF quantization levels used by llama.cpp, Ollama, and LM Studio. The VRAM ratio is relative to the BF16 reference — a 70B model in Q4_K_M, for instance, takes roughly 70 × 0.5 = 35GB of weights (KV cache and context add more).

Most users settle on Q4_K_M for the largest model that fits, then step up to Q5_K_M or Q6_K if they have headroom. Q8 and FP16 are rarely worth it: the quality difference vs Q5 is barely measurable and the memory cost is significant.

GGUF quant names look cryptic but follow a pattern.

• Q4, Q5, Q8 — bits per weight (on average; some weights use more, some less).
• _K — uses k-quants, llama.cpp's modern quantization method (better quality than the legacy Q4_0/Q4_1 scheme).
• _S / _M / _L — small, medium, large variant. Each step up uses ~10% more memory for a small quality gain.
• IQ — i-quants, which use importance matrices to pack bits more intelligently. IQ4_XS often matches Q4_K_M quality at smaller size.
• _0, _1 — legacy (pre-k-quant) variants. Generally superseded by the K variants.

So Q4_K_M reads as "4-bit, k-quant family, medium variant". That's it.

GGUF isn't the only quant format. The format you pick depends on the engine you're running.

• GGUF — used by llama.cpp, Ollama, LM Studio, KoboldCpp. The most portable format. Runs on every backend (CPU, CUDA, ROCm, Metal, Vulkan).
• EXL2 — used by ExLlamaV2 and ExLlamaV3. NVIDIA-only. Often the highest single-stream throughput on CUDA.
• AWQ / GPTQ — used by vLLM and similar serving engines. NVIDIA-first. Tuned for batched multi-user serving.
• MLX quants — used by MLX. Apple Silicon only. Typically the fastest option on a Mac.
• FP8, NVFP4 — newer hardware-accelerated formats on Hopper / Blackwell GPUs. Used by vLLM, TensorRT-LLM. Very fast but limited hardware support.

1. Start from your VRAM. Whatever model you want to run, the weights need to fit with at least ~4–8GB of headroom for context and KV cache.
2. Try Q4_K_M of the largest model that fits. A larger model in Q4 almost always beats a smaller model in Q8. Quality scales with parameters more than with quant level.
3. Step up to Q5 or Q6 if you have headroom. The quality gain is small but free if you're not memory-bound.
4. Don't go below Q4 unless you have to. Q3 and Q2 quants degrade quality noticeably. If a Q4 doesn't fit, prefer a smaller model.
5. Match the format to your engine. GGUF if you're on llama.cpp / Ollama / LM Studio. EXL2 if on ExLlamaV2. AWQ/GPTQ if on vLLM. MLX if on Apple Silicon at the limits of your memory.

• VRAM requirements — how much memory each model size actually needs.
• Best GPUs for local LLMs — ranked by community reports.
• Best GPUs for 70B models — what hardware actually runs them.
• All inference engines — compare llama.cpp, vLLM, MLX, ExLlamaV2.