Local LLM inference engines
The runtime you pick matters as much as the GPU. Each engine has its own strengths: single-user latency, batched throughput, hardware coverage, quantization support. Below: every engine with community performance reports on llamaperf.
- llama.cpp1014 reports
The C/C++ inference runtime behind the GGUF format for quantized open-weight LLMs.
- vLLM207 reports
An inference server with PagedAttention, built for serving many requests at once.
- Strata151 reports
Strata is a local inference runtime for Qwen3.8-Flash-Next.
- MLX107 reports
Apple's machine-learning framework for Apple Silicon, with its own LLM runtime, mlx-lm.
- Ollama84 reports
A local model runner built on llama.cpp's GGML library, with a model registry and a one-line install.
- oMLX65 reports
An inference engine for running open-weight LLMs locally.
- LM Studio60 reports
A desktop app for running local LLMs, with a chat window, a model browser and a local API server.
- NInfer52 reports
An inference engine for running open-weight LLMs locally.
- Gufo50 reports
An inference engine for running open-weight LLMs locally.
- SGLang43 reports
An inference engine for running open-weight LLMs locally.
- ExLlamaV327 reports
An inference engine for running open-weight LLMs locally.
- Inco Splash20 reports
An inference engine for running open-weight LLMs locally.
- ds420 reports
An inference engine for running open-weight LLMs locally.
- ik_llama.cpp19 reports
An inference engine for running open-weight LLMs locally.
- TensorSharp14 reports
An inference engine for running open-weight LLMs locally.
- mlx-serve14 reports
An inference engine for running open-weight LLMs locally.
- mlx-vlm11 reports
An inference engine for running open-weight LLMs locally.
- FreeToken10 reports
An inference engine for running open-weight LLMs locally.
- Unsloth Studio8 reports
An inference engine for running open-weight LLMs locally.
- TabbyAPI3 reports
An inference engine for running open-weight LLMs locally.
- text-generation-webui3 reports
An inference engine for running open-weight LLMs locally.
- KoboldCpp2 reports
An inference engine for running open-weight LLMs locally.
- Paddock2 reports
An inference engine for running open-weight LLMs locally.
- Transformers2 reports
An inference engine for running open-weight LLMs locally.
- mlx-dspark2 reports
An inference engine for running open-weight LLMs locally.
- vllm-mlx2 reports
An inference engine for running open-weight LLMs locally.
- ExLlamaV2no data yet
An NVIDIA-focused runtime for EXL2-quantized models, designed for single-stream speed on one GPU.