Best NVIDIA GPUs for local LLMs

NVIDIA still has the deepest software stack for local inference: CUDA kernels in every major engine, exllamav2 throughput, vLLM batching. The ranking below is from real community submissions, not synthetic benchmarks.

Ranked from 369 community reports on llamaperf.

Ranked by community reports

#	GPU	VRAM	Reports	Fastest t/s
1	RTX 3090nvidia	24GB	87	382.0
2	RTX 5090nvidia	32GB	67	578.0
3	RTX Pro 6000 Blackwellnvidia	96GB	27	177.0
4	RTX 5060 Ti 16GBnvidia	16GB	26	76.7
5	DGX Sparknvidia	128GB	25	181.0
6	RTX 3060 12GBnvidia	12GB	24	70.0
7	RTX 4090nvidia	24GB	17	180.0
8	RTX 5070 Tinvidia	16GB	14	115.0
9	RTX 5080nvidia	16GB	7	75.0
10	RTX PRO 6000 Max-Qnvidia	96GB	6	240.0
11	V100 32GBnvidia	32GB	6	218.0
12	RTX 3080 20GBnvidia	20GB	6	57.5
13	RTX 4060 Ti 16GBnvidia	16GB	6	32.5
14	CMP 170HXnvidia	8GB	5	210.0
15	H100 80GBnvidia	80GB	5	193.0
16	RTX 6000nvidia	48GB	5	150.0
17	RTX 4070 Ti Supernvidia	16GB	5	110.2
18	RTX 4070nvidia	12GB	4	55.0
19	RTX 5070 Ti Laptop 12GBnvidia	12GB	3	59.0
20	RTX 4080nvidia	16GB	3	56.5
21	H200nvidia	141GB	3	4.8
22	V100 16GBnvidia	16GB	2	219.1
23	RTX 2080 Tinvidia	11GB	2	45.0
24	RTX 5070nvidia	12GB	2	22.0
25	RTX A6000 48GBnvidia	48GB	2	17.2
26	RTX 4050 6GBnvidia	6GB	1	129.0
27	RTX 3090 Tinvidia	24GB	1	100.0
28	RTX 4080 Supernvidia	16GB	1	59.0
29	A100 80GBnvidia	80GB	1	56.8
30	RTX Pro 4500 Blackwell 32GBnvidia	32GB	1	45.2
31	RTX 3060 Laptop 6GBnvidia	6GB	1	30.0
32	NVIDIA P102-100nvidia	10GB	1	23.5
33	T4 16GBnvidia	16GB	1	17.6
34	A100 40GBnvidia	40GB	1	16.1
35	L4nvidia	24GB	1	—

What to look for

Used 3090s remain the price/performance king

A used RTX 3090 with 24GB VRAM consistently runs 30B-class models at usable speeds and can handle 70B with two cards in tensor-parallel mode. Despite being two generations old, the bandwidth (936 GB/s) is competitive with new cards costing 3× as much.

RTX 4090 vs RTX 3090

On pure inference (memory-bandwidth-bound), the 4090 is only ~10–15% faster than the 3090 despite massively more compute. The 4090 wins on prompt processing (compute-bound) and any workload involving training/fine-tuning, but for dollar-per-token-per-second, the 3090 is still hard to beat.

Workstation cards (A6000, A100, H100)

These give you 48–80GB of VRAM in a single card — enough for 70B+ models without partitioning. The premium over consumer cards is steep but justified for production serving or research workloads where multi-card scaling adds latency overhead.

Frequently asked

What is the best NVIDIA GPU for local LLMs?

For most users, an RTX 4090 (new) or RTX 3090 (used) — both 24GB — hit the sweet spot for 30B-class models. For 70B work, an RTX A6000 (48GB) or two 3090s in tensor-parallel mode are the standard recommendations.

Is an RTX 4090 worth it over an RTX 3090 for local LLMs?

For pure inference, the 4090 is only modestly faster (~10–15%) because both are limited by memory bandwidth on the same model. The 4090 wins clearly on prompt processing speed and any compute-bound workload (training, fine-tuning, very long contexts).

Can I run local LLMs on an RTX 3060 or 4060?

Yes, on smaller models. A 12GB card runs 7B–13B models comfortably at Q4 quantization. 8GB cards are workable for 7B but tight on context length.

How we rank

Hardware is sorted by the number of community submissions on llamaperf — a proxy for how widely each card is used in practice for local LLM inference. Within that, we surface the fastest tokens-per-second observed on each as a quality signal. Submissions come primarily from r/LocalLLaMA discussions and direct user uploads. Nothing here is sponsored or affiliate-driven.