Best NVIDIA GPUs for local LLMs
NVIDIA cards run every major inference engine, so the choice comes down to memory size and memory speed. This page compares GeForce, workstation and data center cards on the same model at the same quant, from plain single-GPU runs people have reported.
What is the best NVIDIA GPU for local LLMs?
Measured on Qwen3.8 27B at 4-bit, the fastest card people have reported is NVIDIA RTX 5090, at 80.0 tokens per second (median of 7 runs). NVIDIA RTX Pro 6000 Blackwell follows at 67.0 tokens per second (one run), then NVIDIA RTX 5070 Ti at 54.7 tokens per second (one run). The table lists every card with a run of this model.
- Up to 16 GBNVIDIA RTX 5070 TiGen 54.7 t/s
- 17-24 GBNVIDIA RTX 4090Gen 40.0 t/s
- 25-32 GBNVIDIA RTX 5090Gen 80.0 t/s
- 33-96 GBNVIDIA RTX Pro 6000 BlackwellGen 67.0 t/s
- Over 96 GBNVIDIA DGX SparkPP 899.9 t/s
Speed only matters if the model fits. Check what fits on your card.
Ranked on Qwen3.8 27B at 4-bit
Median tokens per second from plain runs: one GPU, one request, no speculative decoding, and the whole model in its memory. A run that kept part of the model in system RAM or on disk isn't ranked, because then the CPU and its memory set the pace as much as the GPU. The runs still differ in engine and context, and a long context can cut decode speed sharply, so each row shows what its median came from. Read a small gap between two rows as noise. Every Qwen3.8 report
| # | GPU | Memory | Gen t/s | PP t/s | Gen runs | Engine and context |
|---|---|---|---|---|---|---|
| 1 | NVIDIA RTX 5090 | 32 GB | 80.0 | - | 7 | llama.cpp, TensorSharp +1, 2 not stated40K to 256K context, 2 not stated |
| 2 | NVIDIA RTX Pro 6000 Blackwell | 96 GB | 67.0 | - | 1 | engine not statedcontext not stated |
| 3 | NVIDIA RTX 5070 Ti | 16 GB | 54.7 | - | 1 | llama.cpp110K context |
| 4 | NVIDIA RTX 4090 | 24 GB | 40.0 | 1949.32 runs | 3 | llama.cpp8K to 194K context |
| 5 | NVIDIA RTX 3090 | 24 GB | 38.4 | 561.02 runs | 5 | llama.cpp, vLLM128K to 256K context, 1 not stated |
| 6 | NVIDIA RTX 5090 Laptop 24GB | 24 GB | 33.5 | - | 2 | llama.cpp, 1 not stated64K to 100K context |
| 7 | NVIDIA V100 32GB | 32 GB | 31.4 | 940.52 runs | 2 | llama.cpp, 1 not stated32K to 128K context |
| 8 | NVIDIA RTX 3080 20GB | 20 GB | 25.7 | 997.01 run | 2 | llama.cpp, 1 not stated96K context, 1 not stated |
| 9 | NVIDIA L4 | 24 GB | 14.6 | - | 1 | llama.cppcontext not stated |
| 10 | NVIDIA RTX 4070 Ti Super | 16 GB | 5.7 | - | 1 | llama.cpp4K context |
| 11 | NVIDIA DGX Spark | 128 GBunified | - | 899.91 run | 0 | llama.cpp16K context |
No speed reports yet
These match the profile, but nobody has reported a plain run on them.
What to look for
Older cards with fast memory hold up
Generation speed follows memory bandwidth, and a card from two generations back can have nearly as much as a new one. That's why used 24 GB GeForce cards stay popular for local models. Check the table before you pay for a newer generation.
Compute matters for long prompts
Reading a long prompt is compute-bound, so a newer card pulls ahead there even when generation speed is close. If you paste long documents or run coding agents with big contexts, look at prompt speed on the card's page as well.
Workstation and data center cards
These carry far more memory on one card, which lets you run a large model without splitting it across GPUs. You pay a lot more for that memory, so compare them against two consumer cards before you buy.
Frequently asked
Is a new NVIDIA card worth it over a used one for local LLMs?
For generation speed, often not by much, because both are limited by memory bandwidth. A newer card wins on prompt processing and on any training or fine-tuning. Compare the two cards' medians in the table on the same model.
Can I run local LLMs on an 8 GB or 12 GB NVIDIA card?
Yes, with smaller models or smaller quants. The VRAM calculator lists what fits on your exact card, and the coding pages rank models by memory size.
How is this list ranked?
Every card is ranked on its median speed for one reference model at one quant level, counting plain single-GPU runs only. The reference is the model the most NVIDIA cards have reports of.