llamaperf

Best local LLMs for 24GB VRAM

How far 24GB of graphics memory goes for local models on cards like NVIDIA RTX 3090, NVIDIA RTX 4090, AMD RX 7900 XTX or NVIDIA Tesla P40 24GB: which models fit, and what people actually run on it.

Is 24GB of VRAM enough?

24GB holds a dense model of up to about 28B parameters at Q4_K_M with an 8K context, by the calculator's memory estimate. The largest model people report here that fits comfortably is Gemma 4 31B, which needs about 19.8 GB at Q4_K_M with an 8K context. Anything bigger needs a smaller quant, part of the model in system RAM, or more memory, and a longer context needs more room on top.

The calculator checks your exact card, including longer contexts and offloading.

Models that fit in 24GB

Models people report on llamaperf, largest first, at the best quant whose memory with an 8K context stays within the calculator's comfortable band (85% of the card). Estimated from each model's shape.

And 106 smaller models. The calculator lists them all for your card.

What people run on 24GB

Model sizes reported on setups with more than 16 GB and up to 24 GB of memory, most reported first. Typical speed is the median of plain runs, and speculative runs are counted apart.

#ModelReports
1Qwen3.827B
Alibaba · mostly Q4_K_M
Gen 36 t/s (37 runs) · PP 775 t/s (16 runs) · with speculation 77 t/s (53)
160
2Qwen3.8 Flash-Next125B · 6B active
Alibaba · mostly IQ3_XXS
No generation data · with speculation 86 t/s (1)
67
3Qwen3.627B
Alibaba · mostly Q4_K_M
Gen 35 t/s (4 runs) · PP 616 t/s (3 runs) · with speculation 50 t/s (6)
20
4Qwen3.635B · 3B active
Alibaba · mostly Q4_K_M
Gen 111 t/s (8 runs) · PP 1428 t/s (2 runs) · with speculation 52 t/s (1)
18
5DeepSeek V4 Flash284B · 13B active
DeepSeek · mostly IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8
No generation data
16
6Qwen3.8 Swift-1.527B
Alibaba · mostly IQ3_S
Gen 67 t/s (1 run) · with speculation 80 t/s (4)
10
7DeepSeek V4.1 Flash552B · 16B active
mostly UD-IQ2_XXS
No generation data
9
8Gemma 426B · 4B active
Google DeepMind · mostly Q4_0
Gen 143 t/s (5 runs) · PP 4966 t/s (2 runs)
7
9GLM-5.3 Flash320B · 18B active
Zhipu AI · mostly 3.05bpw
No generation data
6
10Gemma 4 E4B8B
Google DeepMind · mostly Q4_K_M
Gen 64 t/s (2 runs) · PP 83 t/s (1 run)
5
11Gemma 431B
Google DeepMind · mostly Q4_K_S
Gen 37 t/s (3 runs) · PP 422 t/s (1 run)
5
12Muse Glimmer30B
Meta · mostly IQ4_XS
Gen 30 t/s (1 run) · with speculation 94 t/s (3)
5
13Qwen3.527B
Alibaba · mostly Q4_K_M
Gen 42 t/s (1 run)
3
14Qwen3.8 Uncensored27B
Alibaba · mostly Q4_K_M
No generation data · with speculation 20 t/s (1)
3
15Qwen30.6B
Alibaba · mostly Q4
Gen 531 t/s (1 run)
2
16LFM31.7B
mostly Q8_0
Gen 340 t/s (2 runs) · PP 20324 t/s (2 runs)
2
17Nemotron 3.5 Bolt74B · 9B active
mostly 2-bit
Gen 86 t/s (2 runs) · PP 828 t/s (2 runs)
2
18Qwen2.57B
Alibaba · mostly GPTQ-Int4
Gen 66 t/s (2 runs)
2
19Gemma 412B
Google DeepMind · mostly QAT
Gen 48 t/s (2 runs) · PP 139 t/s (1 run)
2
20Qwen3.59B
Alibaba · mostly BF16
Gen 29 t/s (2 runs)
2
21Qwen314B
Alibaba · mostly 4-bit
Gen 12 t/s (1 run)
2
22Qwen3.535B · 3B active
Alibaba · mostly IQ4_NL
Gen 8.3 t/s (1 run) · PP 546 t/s (2 runs)
2
23Nemotron 3.5 Lightning30B · 3B active
NVIDIA · mostly Q4_0
No generation data
2
24Ornith1.5122B
mostly Q4_K_XL
No generation data
2
25Qwen3-Next80B · 3B active
Alibaba · mostly Q4_K_M
No generation data
2
26Qwen3.8 Swift-1.5-Uncensored27B
Alibaba · mostly IQ4_XS
No generation data · with speculation 68 t/s (1)
2
27Qwen2.5 Coder-7B7B
Alibaba · mostly Q4_K_M
Gen 159 t/s (1 run)
1
28DeepSeek-Coder-V2-Lite16B · 2.4B active
mostly 4bit
Gen 126 t/s (1 run) · PP 364 t/s (1 run)
1
29Llama 38B
mostly Q4_K_M
Gen 112 t/s (1 run) · PP 3865 t/s (1 run)
1
30Inkling 2 Mini26B · 3.4B active
mostly UD-Q4_K_M
Gen 109 t/s (1 run) · PP 2075 t/s (1 run)
1
31Qwen3-Coder30B · 3B active
mostly Q4_K_M
Gen 107 t/s (1 run) · PP 3707 t/s (1 run)
1
32Qwen330B · 3B active
Alibaba · mostly IQ3_XXS
Gen 74 t/s (1 run)
1
33Qwen3.8 Escha-W227B
Alibaba · mostly escha 2-bit
Gen 67 t/s (1 run)
1
34Ornith1.535B · 3B active
mostly Q4_0
Gen 61 t/s (1 run) · PP 1104 t/s (1 run)
1
35Inkling 2 Mini abliterated26B · 3B active
mostly XQ
Gen 60 t/s (1 run) · PP 1162 t/s (1 run)
1
36Qwen3.8 Jev27B
Alibaba · mostly 4.0bpw
Gen 54 t/s (1 run) · PP 885 t/s (1 run)
1
37Llama 3.23B
Meta · mostly Q4_K_M
Gen 46 t/s (1 run) · PP 530 t/s (1 run)
1
38Qwen3.835B · 3B active
Alibaba · mostly IQ3_M
Gen 45 t/s (1 run)
1
39Qwen3.8 AREX-227B
Alibaba · mostly EXL3 4.00 bpw
Gen 36 t/s (1 run)
1
40Qwen2.5 Coder-32B32B
Alibaba · mostly Q4_K_M
Gen 28 t/s (1 run) · PP 450 t/s (1 run)
1
41Qwen38B
Alibaba · mostly Q4_K_M
Gen 25 t/s (1 run)
1
42Qwen 2.5 Coder7B
mostly Q4_K_M
Gen 23 t/s (1 run) · PP 240 t/s (1 run)
1
43Qwen3.8 27B dense27B
Alibaba · mostly 4bit
Gen 9.8 t/s (1 run)
1
44Raon-1 mini-preview29B · 4B active
mostly MQ4
Gen 4.6 t/s (1 run) · PP 13 t/s (1 run)
1
45Qwen3 Heretic14B
Alibaba · mostly INT4_SYM
Gen 3.6 t/s (1 run)
1
46Bonsai 2 Ternary27B
PrismML · mostly Q4/Q5
No generation data · with speculation 188 t/s (1)
1
47DeepSeek V4 Flash Vision-Exp285B · 13B active
DeepSeek · mostly FP4
No generation data
1
48Edge0145B
mostly q3k
No generation data
1
49GLM-4.5-Air106B · 12B active
No generation data
1
50GLM-5.2744B · 40B active
Zhipu AI · mostly UD-IQ2_M
No generation data
1
51Gemma 31B
Google
No generation data
1
52Gemma 4 Uncensored-HauhauCS-Balanced26B · 4B active
Google DeepMind · mostly Q4_K_M
No generation data
1
53Gemma 426B
Google DeepMind
No generation data
1
54Nemotron 3 Super120B · 12B active
mostly ~3 bits per weight
No generation data
1
55Ornith1.5 BigBang35B · 3B active
mostly Q4_K_M
No generation data · with speculation 169 t/s (1)
1
56Ornith1.5 Ornith-1.5-35B-A3B35B · 3B active
mostly IQ4_XS
No generation data
1
57Qwen2.532B
Alibaba · mostly AWQ
No generation data
1
58Qwen3235B · 22B active
Alibaba · mostly Q4_K_M
No generation data
1
59Qwen3.54B
Alibaba
No generation data
1
60Qwen3.5 Mica v0.14B
Alibaba · mostly Q4_K_M
No generation data
1
61Qwen3.6 Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-LOW27B
Alibaba · mostly IQ4_XS
No generation data · with speculation 42 t/s (1)
1
62Qwen3.6 Heretic-v227B
Alibaba · mostly Q4_K_M
No generation data · with speculation 80 t/s (1)
1
63Qwen3.6 Opus distill, abliterated35B · 3B active
Alibaba · mostly i1-IQ3_XXS
No generation data
1
64Qwen3.8 Bonsai 227B
Alibaba · mostly PQ2_0
No generation data · PP 835 t/s (1 run)
1
65Qwen3.8 Qwen3.8-27B27B
Alibaba · mostly Q4_K_S
No generation data
1
66Qwen3.8 Uncensored-HauHauCS27B
Alibaba · mostly INT8
No generation data
1
67Qwen3.8 Defiance-Uncensored27B
Alibaba · mostly IQ4_XS
No generation data
1
68Qwen3.8 pi27B
Alibaba · mostly Q4_K_M
No generation data
1
69Qwen3.8 Swift Uncensored27B
Alibaba · mostly IQ4_XS-M
No generation data
1
70Qwen3.8 Concept27B
Alibaba · mostly W4A16 AutoRound
No generation data
1
71Qwen3.8 Uncensored-HauhauCS-Aggressive27B
Alibaba · mostly IQ4_XS
No generation data · with speculation 72 t/s (1)
1
72Qwen3.8 Huihui-Abliterated27B
Alibaba · mostly Q4_K_S
No generation data
1
73Qwen3.8 Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-NEO27B
Alibaba · mostly Q8_0
No generation data
1
74Qwen3.8 Swift-1.5-HyperQwen27B
Alibaba · mostly W4A16
No generation data
1
75Qwen3.8 CODER27B
Alibaba · mostly IQ4_XS
No generation data
1
76Qwen3.8 Swift27B
Alibaba · mostly IQ4_XS
No generation data
1
77Qwen3.8 heretic-ara27B
Alibaba · mostly Q5_K_M
No generation data
1
78Qwen3.8 Hui Hui27B
Alibaba · mostly Q4
No generation data
1
79Qwen3.8 Swift-1.5-125B-A6B125B · 6B active
Alibaba · mostly IQ3_S
No generation data
1
80Qwen3.8 Swift-1.5 Flash-Next125B · 6B active
Alibaba · mostly IQ2_XS
No generation data
1
81gpt-oss21B · 3.6B active
OpenAI
No generation data
1

Frequently asked

What do people run on 24GB of VRAM?

On setups with more than 16 GB and up to 24 GB of memory, the model people report most is Qwen3.8 27B, followed by Qwen3.8 125B · 6B active and Qwen3.6 27B. The table lists each with the quant most people used and the typical speed.

Does a Mac with 24GB count as 24GB of VRAM?

No. macOS keeps part of a Mac's unified memory back from the GPU, so a Mac with 24GB can use less than that for a model. The calculator's Mac tab grades every Mac against its real limit.