Best GPUs for running local LLMs
The best GPU for you is the fastest one with enough memory for the models you want to run. This page compares cards on the same model at the same quant, using plain single-GPU runs people have reported. Those runs still used different engines and context lengths, so each card's row says which. Treat a small gap between two cards as noise rather than a verdict.
What is the best GPU for running LLMs locally?
Measured on Qwen3.8 27B at 4-bit, the fastest card people have reported is NVIDIA RTX 5090, at 80.0 tokens per second (median of 7 runs). NVIDIA RTX Pro 6000 Blackwell follows at 67.0 tokens per second (one run), then NVIDIA RTX 5070 Ti at 54.7 tokens per second (one run). The table lists every card with a run of this model.
- Up to 16 GBNVIDIA RTX 5070 TiGen 54.7 t/s
- 17-24 GBAMD RX 7900 XTXGen 42.5 t/s
- 25-32 GBNVIDIA RTX 5090Gen 80.0 t/s
- 33-96 GBNVIDIA RTX Pro 6000 BlackwellGen 67.0 t/s
- Over 96 GBM5 Ultra 256GBGen 50.0 t/s
Speed only matters if the model fits. Check what fits on your card.
Ranked on Qwen3.8 27B at 4-bit
Median tokens per second from plain runs: one GPU, one request, no speculative decoding, and the whole model in its memory. A run that kept part of the model in system RAM or on disk isn't ranked, because then the CPU and its memory set the pace as much as the GPU. The runs still differ in engine and context, and a long context can cut decode speed sharply, so each row shows what its median came from. Read a small gap between two rows as noise. Every Qwen3.8 report
| # | GPU | Memory | Gen t/s | PP t/s | Gen runs | Engine and context |
|---|---|---|---|---|---|---|
| 1 | NVIDIA RTX 5090 | 32 GB | 80.0 | - | 7 | llama.cpp, TensorSharp +1, 2 not stated40K to 256K context, 2 not stated |
| 2 | NVIDIA RTX Pro 6000 Blackwell | 96 GB | 67.0 | - | 1 | engine not statedcontext not stated |
| 3 | NVIDIA RTX 5070 Ti | 16 GB | 54.7 | - | 1 | llama.cpp110K context |
| 4 | M5 Ultra 256GB | 256 GBunified | 50.0 | 1800.01 run | 1 | oMLX8K context |
| 5 | AMD RX 7900 XTX | 24 GB | 42.5 | 640.04 runs | 6 | llama.cpp, Ollama, 2 not stated64K to 220K context, 3 not stated |
| 6 | NVIDIA RTX 4090 | 24 GB | 40.0 | 1949.32 runs | 3 | llama.cpp8K to 194K context |
| 7 | NVIDIA RTX 3090 | 24 GB | 37.4 | 561.02 runs | 6 | llama.cpp, vLLM, 1 not stated128K to 256K context, 1 not stated |
| 8 | AMD RX 7800 XT 16GB | 16 GB | 35.0 | - | 2 | llama.cpp, 1 not stated100K to 120K context |
| 9 | NVIDIA RTX 5090 Laptop 24GB | 24 GB | 33.5 | - | 2 | llama.cpp, 1 not stated64K to 100K context |
| 10 | M3 Ultra 512GB | 512 GBunified | 31.8 | 391.21 run | 1 | MLXcontext not stated |
| 11 | NVIDIA V100 32GB | 32 GB | 31.4 | 940.52 runs | 2 | llama.cpp, 1 not stated32K to 128K context |
| 12 | AMD Radeon AI PRO R9700 32GB | 32 GB | 29.8 | 1287.01 run | 3 | llama.cpp, 1 not stated2K context, 2 not stated |
| 13 | M5 Max 36GB | 36 GBunified | 25.8 | - | 1 | engine not statedcontext not stated |
| 14 | NVIDIA RTX 3080 20GB | 20 GB | 25.7 | 997.01 run | 2 | llama.cpp, 1 not stated96K context, 1 not stated |
| 15 | Intel Arc Pro B70 | 32 GB | 25.6 | 512.11 run | 1 | llama.cppcontext not stated |
| 16 | M1 Ultra 128GB | 128 GBunified | 25.1 | 153.31 run | 1 | engine not statedcontext not stated |
| 17 | M4 Max 64GB | 64 GBunified | 23.9 | 119.91 run | 1 | llama.cppcontext not stated |
| 18 | M4 Max 128GB | 128 GBunified | 21.8 | 236.61 run | 1 | llama.cppcontext not stated |
| 19 | Intel Arc Pro B60 24GB | 24 GB | 18.8 | 339.43 runs | 4 | llama.cpp, 1 not stated512 to 150K context, 1 not stated |
| 20 | M1 Max 64GB | 64 GBunified | 16.9 | - | 1 | LM Studio64K context |
| 21 | M1 Max 32GB | 32 GBunified | 15.8 | 81.81 run | 1 | MLXcontext not stated |
| 22 | NVIDIA L4 | 24 GB | 14.6 | - | 1 | llama.cppcontext not stated |
| 23 | M3 Ultra 256GB | 256 GBunified | 14.0 | 93.11 run | 1 | Ollamacontext not stated |
| 24 | AMD Strix Halo 128GB | 128 GBunified | 13.1 | 378.23 runs | 4 | llama.cpp, 1 not stated32K context, 3 not stated |
| 25 | M4 Pro 64GB | 64 GBunified | 12.9 | 126.24 runs | 4 | llama.cpp, 2 not statedcontext not stated |
| 26 | M2 Pro 32GB | 32 GBunified | 8.6 | 21.91 run | 1 | llama.cpp128K context |
| 27 | M4 24GB | 24 GBunified | 6.5 | 60.81 run | 1 | llama.cppcontext not stated |
| 28 | NVIDIA RTX 4070 Ti Super | 16 GB | 5.7 | - | 1 | llama.cpp4K context |
| 29 | NVIDIA DGX Spark | 128 GBunified | - | 899.91 run | 0 | llama.cpp16K context |
No speed reports yet
These match the profile, but nobody has reported a plain run on them.
What to look for
Memory decides what runs
A model either fits in your card's memory or it doesn't. When it doesn't, the engine spills layers to system RAM and generation slows to a crawl, or the load fails. Check the size you want in the VRAM calculator before you compare speeds.
Bandwidth decides how fast it generates
Once the weights fit, each new token means reading them from memory again. That's why an older card with fast memory can keep up with a much newer one, and why the ranking below follows memory bandwidth more closely than compute.
Software support decides which engines you can use
NVIDIA cards run every major engine. AMD cards run llama.cpp and vLLM through ROCm and Vulkan, with fewer options and more setup. Macs run llama.cpp and MLX well and can't run vLLM. Pick the engine you want first, then the hardware that runs it.
Frequently asked
Is more VRAM or a faster GPU better for local LLMs?
More VRAM, as long as the card's memory is fast. A model that fits entirely on the card runs many times faster than one that spills into system RAM, so a slower card with room for the whole model beats a faster card that has to offload.
Do I need an NVIDIA GPU for local LLMs?
No. AMD cards and Apple Silicon Macs both run local models, and the table ranks them alongside NVIDIA where people have reported the same model. NVIDIA still has the widest engine support and the least setup.
Can I use two GPUs for a local LLM?
Yes. llama.cpp, vLLM and ExLlama can split a model across cards, which is the usual way to run a model too big for one. This page ranks single-GPU runs only, because a split run's speed depends on how the cards are linked as well as on the cards.
How is this list ranked?
Every card is ranked on its median speed for one reference model at one quant level, counting plain runs only: one GPU, one request, no speculative decoding, and the whole model in the card's memory. The reference is the model the most cards have reports of, so it changes as people move to newer models.