Choosing a model
Ollama, llama.cpp, vLLM or MLX-LM: picking an engine
By llamaperf · · 5 min read
Quick answer
Pick the engine by how you'll use the model. For one person on a PC or Mac, Ollama or llama.cpp is the easy place to start. Serving lots of requests from NVIDIA or AMD GPUs is what vLLM is for, and on Apple silicon MLX-LM is worth trying next to llama.cpp. After that, run the same model and prompt on two of them and keep whichever fits your routine.
Start from how you'll use it
Choose for the people and applications that will send requests, then check engine benchmarks.
Just you, on your own machine
As the only user, you want quick setup and good speed on consumer hardware, and Ollama and llama.cpp both give you that. Ollama wraps a model library and a simple API around the engine. llama.cpp hands you the engine itself, with every setting exposed. If you're on a Mac, add MLX-LM to the shortlist.
Other people or programs, all day
When a team or an app calls the model constantly, what you care about is how many requests it can handle together before each one starts to crawl. Serving engines like vLLM are designed for that. Our post on sharing one model with a few people covers the point up to which a small group can do without one.
What each engine is built for
llama.cpp and Ollama
llama.cpp runs on almost anything, from a laptop CPU to a multi-GPU box, and the first item in its README's feature list is this:
Plain C/C++ implementation without any dependencies
From the llama.cpp README
That's why so many other tools use it underneath. Ollama reads the same GGUF format and is one of the easiest ways to get a model running, since you pull a model by name. When its library doesn't have what you want, the Ollama docs on importing models show how to bring in your own GGUF or safetensors files.
vLLM and MLX-LM
vLLM is aimed at serving, and its README introduces it like this:
vLLM is a fast and easy-to-use library for LLM inference and serving.
From the vLLM README
The same README lists continuous batching and an OpenAI-compatible API server among its features. MLX-LM is the MLX project's package for running and fine-tuning models on Apple silicon, and it pulls ready-converted models from the MLX Community organisation on Hugging Face.
Model formats decide a lot
Choosing an engine also chooses which file formats you can load, and the format decides how much memory the model needs.
GGUF and its quantizations
llama.cpp and Ollama use GGUF files, which come in many quantization levels. In the llama.cpp quantize README, Llama 3.1 8B is 14.96 GiB in F16 and 4.58 GiB at Q4_K_M, less than a third of the size, and that shrinking is how big models end up fitting on ordinary hardware. Our post on what Q4_K_M means explains the names, and the quantization guide goes through the trade-offs.
Check what your hardware runs
Serving engines usually want the model in its Hugging Face form or in a quantization built for GPUs, so the same model can need very different amounts of memory depending on the engine. On a Mac, MLX-LM loads MLX-converted weights and llama.cpp loads GGUF, and both run on Apple's GPU. The calculator shows the memory a given quantization needs on your card, and our Mac pages show what people run on each chip.
Try two before you commit
Use the same model and prompt
Get one model in the format each engine uses, at a similar quantization. Keep the prompt and context size identical and cap the output at the same length, then time the wait before the first word and the whole answer on both. Because engines report speed differently, use your own consistent timing method where possible.
Keep the one that fits your routine
Check how the engine handles updates and model downloads. It also needs to support the API your tools expect and let you change settings without getting in your way. The engine pages show what other people run on hardware like yours. Once you've settled, submit a report with the engine name and version in it, since two people on the same card and model can see different speeds just from running different engines.
Frequently asked questions
Should I use Ollama or llama.cpp?
Ollama if you want quick setup and models managed for you, llama.cpp if you want every setting exposed and new features first. They run the same GGUF files, so starting with one doesn't lock you out of the other.
When should I use vLLM instead of Ollama?
Once lots of requests hit the model at the same time, say a team or an app calling an API all day. vLLM is built for that kind of serving, with continuous batching. For just you, Ollama or llama.cpp is simpler.
Is MLX faster than llama.cpp on a Mac?
It depends on the model and the quantization, and both run on the Mac's GPU. Run the same model in each format with the same prompt on your own machine and compare the times.
Can Ollama run GGUF files from Hugging Face?
Yes. The Ollama docs describe importing one through a Modelfile that points at the GGUF file, and the same page covers safetensors weights.
Does vLLM use less VRAM than llama.cpp?
Not as a rule. Memory mostly depends on the format and quantization each engine loads, plus the context and how many requests run in parallel, so compare them at matched settings.
Can I switch engines without downloading the model again?
Only if the new engine reads the same format. llama.cpp and Ollama can share one GGUF file, while vLLM and MLX-LM usually need a different download or a conversion.