Best local LLMs for 24GB VRAM
How far 24GB of graphics memory goes for local models on cards like NVIDIA RTX 3090, NVIDIA RTX 4090, AMD RX 7900 XTX or NVIDIA Tesla P40 24GB: which models fit, and what people actually run on it.
Is 24GB of VRAM enough?
24GB holds a dense model of up to about 28B parameters at Q4_K_M with an 8K context, by the calculator's memory estimate. The largest model people report here that fits comfortably is Gemma 4 31B, which needs about 19.8 GB at Q4_K_M with an 8K context. Anything bigger needs a smaller quant, part of the model in system RAM, or more memory, and a longer context needs more room on top.
The calculator checks your exact card, including longer contexts and offloading.
Models that fit in 24GB
Models people report on llamaperf, largest first, at the best quant whose memory with an 8K context stays within the calculator's comfortable band (85% of the card). Estimated from each model's shape.
| Model | Best quant | Needs |
|---|---|---|
| Qwen3.6 35B · 3B active | Q3_K_M | 17.2 GB |
| Qwen3.5 35B · 3B active | Q3_K_M | 17.2 GB |
| Ornith1.5 35B · 3B active | Q3_K_M | 17.2 GB |
| MiniCPM-V 4.7 35B · 3B active | Q3_K_M | 17.2 GB |
| Agents-A1 35B · 3B active | Q3_K_M | 17.2 GB |
| KAT-Coder V2.5-Dev 35B · 3B active | Q3_K_M | 17.2 GB |
| Nex-N2.5-mini 35B · 3B active | Q3_K_M | 17.2 GB |
| Ornith 1.0 35B · 3B active | Q3_K_M | 17.2 GB |
| Qwen3.8 35B · 3B active | Q3_K_M | 17.6 GB |
| Qwen3.8 35B | Q3_K_M | 17.6 GB |
| Nanbeige4.3 33B | Q3_K_M | 18.4 GB |
| Qwen3 32B | Q3_K_M | 17.9 GB |
And 106 smaller models. The calculator lists them all for your card.
What people run on 24GB
Model sizes reported on setups with more than 16 GB and up to 24 GB of memory, most reported first. Typical speed is the median of plain runs, and speculative runs are counted apart.
| # | Model | Reports | Gen t/s | PP t/s | With speculation |
|---|---|---|---|---|---|
| 1 | Qwen3.827B Alibaba · mostly Q4_K_M Gen 36 t/s (37 runs) · PP 775 t/s (16 runs) · with speculation 77 t/s (53) | 160 | 36 37 runs, 10 devices | 775 16 runs | 77 53 runs |
| 2 | Qwen3.8 Flash-Next125B · 6B active Alibaba · mostly IQ3_XXS No generation data · with speculation 86 t/s (1) | 67 | no generation data | no PP data | 86 1 run |
| 3 | Qwen3.627B Alibaba · mostly Q4_K_M Gen 35 t/s (4 runs) · PP 616 t/s (3 runs) · with speculation 50 t/s (6) | 20 | 35 4 runs, 3 devices | 616 3 runs | 50 6 runs |
| 4 | Qwen3.635B · 3B active Alibaba · mostly Q4_K_M Gen 111 t/s (8 runs) · PP 1428 t/s (2 runs) · with speculation 52 t/s (1) | 18 | 111 8 runs, 5 devices | 1428 2 runs | 52 1 run |
| 5 | DeepSeek V4 Flash284B · 13B active DeepSeek · mostly IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 No generation data | 16 | no generation data | no PP data | none |
| 6 | Qwen3.8 Swift-1.527B Alibaba · mostly IQ3_S Gen 67 t/s (1 run) · with speculation 80 t/s (4) | 10 | 67 1 run | no PP data | 80 4 runs |
| 7 | DeepSeek V4.1 Flash552B · 16B active mostly UD-IQ2_XXS No generation data | 9 | no generation data | no PP data | none |
| 8 | Gemma 426B · 4B active Google DeepMind · mostly Q4_0 Gen 143 t/s (5 runs) · PP 4966 t/s (2 runs) | 7 | 143 5 runs, 2 devices | 4966 2 runs | none |
| 9 | GLM-5.3 Flash320B · 18B active Zhipu AI · mostly 3.05bpw No generation data | 6 | no generation data | no PP data | none |
| 10 | Gemma 4 E4B8B Google DeepMind · mostly Q4_K_M Gen 64 t/s (2 runs) · PP 83 t/s (1 run) | 5 | 64 2 runs | 83 1 run | none |
| 11 | Gemma 431B Google DeepMind · mostly Q4_K_S Gen 37 t/s (3 runs) · PP 422 t/s (1 run) | 5 | 37 3 runs, 3 devices | 422 1 run | none |
| 12 | Muse Glimmer30B Meta · mostly IQ4_XS Gen 30 t/s (1 run) · with speculation 94 t/s (3) | 5 | 30 1 run | no PP data | 94 3 runs |
| 13 | Qwen3.527B Alibaba · mostly Q4_K_M Gen 42 t/s (1 run) | 3 | 42 1 run | no PP data | none |
| 14 | Qwen3.8 Uncensored27B Alibaba · mostly Q4_K_M No generation data · with speculation 20 t/s (1) | 3 | no generation data | no PP data | 20 1 run |
| 15 | Qwen30.6B Alibaba · mostly Q4 Gen 531 t/s (1 run) | 2 | 531 1 run | no PP data | none |
| 16 | LFM31.7B mostly Q8_0 Gen 340 t/s (2 runs) · PP 20324 t/s (2 runs) | 2 | 340 2 runs | 20324 2 runs | none |
| 17 | Nemotron 3.5 Bolt74B · 9B active mostly 2-bit Gen 86 t/s (2 runs) · PP 828 t/s (2 runs) | 2 | 86 2 runs | 828 2 runs | none |
| 18 | Qwen2.57B Alibaba · mostly GPTQ-Int4 Gen 66 t/s (2 runs) | 2 | 66 2 runs, 2 devices | no PP data | none |
| 19 | Gemma 412B Google DeepMind · mostly QAT Gen 48 t/s (2 runs) · PP 139 t/s (1 run) | 2 | 48 2 runs, 2 devices | 139 1 run | none |
| 20 | Qwen3.59B Alibaba · mostly BF16 Gen 29 t/s (2 runs) | 2 | 29 2 runs, 2 devices | no PP data | none |
| 21 | Qwen314B Alibaba · mostly 4-bit Gen 12 t/s (1 run) | 2 | 12 1 run | no PP data | none |
| 22 | Qwen3.535B · 3B active Alibaba · mostly IQ4_NL Gen 8.3 t/s (1 run) · PP 546 t/s (2 runs) | 2 | 8.3 1 run | 546 2 runs | none |
| 23 | Nemotron 3.5 Lightning30B · 3B active NVIDIA · mostly Q4_0 No generation data | 2 | no generation data | no PP data | none |
| 24 | Ornith1.5122B mostly Q4_K_XL No generation data | 2 | no generation data | no PP data | none |
| 25 | Qwen3-Next80B · 3B active Alibaba · mostly Q4_K_M No generation data | 2 | no generation data | no PP data | none |
| 26 | Qwen3.8 Swift-1.5-Uncensored27B Alibaba · mostly IQ4_XS No generation data · with speculation 68 t/s (1) | 2 | no generation data | no PP data | 68 1 run |
| 27 | Qwen2.5 Coder-7B7B Alibaba · mostly Q4_K_M Gen 159 t/s (1 run) | 1 | 159 1 run | no PP data | none |
| 28 | DeepSeek-Coder-V2-Lite16B · 2.4B active mostly 4bit Gen 126 t/s (1 run) · PP 364 t/s (1 run) | 1 | 126 1 run | 364 1 run | none |
| 29 | Llama 38B mostly Q4_K_M Gen 112 t/s (1 run) · PP 3865 t/s (1 run) | 1 | 112 1 run | 3865 1 run | none |
| 30 | Inkling 2 Mini26B · 3.4B active mostly UD-Q4_K_M Gen 109 t/s (1 run) · PP 2075 t/s (1 run) | 1 | 109 1 run | 2075 1 run | none |
| 31 | Qwen3-Coder30B · 3B active mostly Q4_K_M Gen 107 t/s (1 run) · PP 3707 t/s (1 run) | 1 | 107 1 run | 3707 1 run | none |
| 32 | Qwen330B · 3B active Alibaba · mostly IQ3_XXS Gen 74 t/s (1 run) | 1 | 74 1 run | no PP data | none |
| 33 | Qwen3.8 Escha-W227B Alibaba · mostly escha 2-bit Gen 67 t/s (1 run) | 1 | 67 1 run | no PP data | none |
| 34 | Ornith1.535B · 3B active mostly Q4_0 Gen 61 t/s (1 run) · PP 1104 t/s (1 run) | 1 | 61 1 run | 1104 1 run | none |
| 35 | Inkling 2 Mini abliterated26B · 3B active mostly XQ Gen 60 t/s (1 run) · PP 1162 t/s (1 run) | 1 | 60 1 run | 1162 1 run | none |
| 36 | Qwen3.8 Jev27B Alibaba · mostly 4.0bpw Gen 54 t/s (1 run) · PP 885 t/s (1 run) | 1 | 54 1 run | 885 1 run | none |
| 37 | Llama 3.23B Meta · mostly Q4_K_M Gen 46 t/s (1 run) · PP 530 t/s (1 run) | 1 | 46 1 run | 530 1 run | none |
| 38 | Qwen3.835B · 3B active Alibaba · mostly IQ3_M Gen 45 t/s (1 run) | 1 | 45 1 run | no PP data | none |
| 39 | Qwen3.8 AREX-227B Alibaba · mostly EXL3 4.00 bpw Gen 36 t/s (1 run) | 1 | 36 1 run | no PP data | none |
| 40 | Qwen2.5 Coder-32B32B Alibaba · mostly Q4_K_M Gen 28 t/s (1 run) · PP 450 t/s (1 run) | 1 | 28 1 run | 450 1 run | none |
| 41 | Qwen38B Alibaba · mostly Q4_K_M Gen 25 t/s (1 run) | 1 | 25 1 run | no PP data | none |
| 42 | Qwen 2.5 Coder7B mostly Q4_K_M Gen 23 t/s (1 run) · PP 240 t/s (1 run) | 1 | 23 1 run | 240 1 run | none |
| 43 | Qwen3.8 27B dense27B Alibaba · mostly 4bit Gen 9.8 t/s (1 run) | 1 | 9.8 1 run | no PP data | none |
| 44 | Raon-1 mini-preview29B · 4B active mostly MQ4 Gen 4.6 t/s (1 run) · PP 13 t/s (1 run) | 1 | 4.6 1 run | 13 1 run | none |
| 45 | Qwen3 Heretic14B Alibaba · mostly INT4_SYM Gen 3.6 t/s (1 run) | 1 | 3.6 1 run | no PP data | none |
| 46 | Bonsai 2 Ternary27B PrismML · mostly Q4/Q5 No generation data · with speculation 188 t/s (1) | 1 | no generation data | no PP data | 188 1 run |
| 47 | DeepSeek V4 Flash Vision-Exp285B · 13B active DeepSeek · mostly FP4 No generation data | 1 | no generation data | no PP data | none |
| 48 | Edge0145B mostly q3k No generation data | 1 | no generation data | no PP data | none |
| 49 | GLM-4.5-Air106B · 12B active No generation data | 1 | no generation data | no PP data | none |
| 50 | GLM-5.2744B · 40B active Zhipu AI · mostly UD-IQ2_M No generation data | 1 | no generation data | no PP data | none |
| 51 | Gemma 31B Google No generation data | 1 | no generation data | no PP data | none |
| 52 | Gemma 4 Uncensored-HauhauCS-Balanced26B · 4B active Google DeepMind · mostly Q4_K_M No generation data | 1 | no generation data | no PP data | none |
| 53 | Gemma 426B Google DeepMind No generation data | 1 | no generation data | no PP data | none |
| 54 | Nemotron 3 Super120B · 12B active mostly ~3 bits per weight No generation data | 1 | no generation data | no PP data | none |
| 55 | Ornith1.5 BigBang35B · 3B active mostly Q4_K_M No generation data · with speculation 169 t/s (1) | 1 | no generation data | no PP data | 169 1 run |
| 56 | Ornith1.5 Ornith-1.5-35B-A3B35B · 3B active mostly IQ4_XS No generation data | 1 | no generation data | no PP data | none |
| 57 | Qwen2.532B Alibaba · mostly AWQ No generation data | 1 | no generation data | no PP data | none |
| 58 | Qwen3235B · 22B active Alibaba · mostly Q4_K_M No generation data | 1 | no generation data | no PP data | none |
| 59 | Qwen3.54B Alibaba No generation data | 1 | no generation data | no PP data | none |
| 60 | Qwen3.5 Mica v0.14B Alibaba · mostly Q4_K_M No generation data | 1 | no generation data | no PP data | none |
| 61 | Qwen3.6 Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-LOW27B Alibaba · mostly IQ4_XS No generation data · with speculation 42 t/s (1) | 1 | no generation data | no PP data | 42 1 run |
| 62 | Qwen3.6 Heretic-v227B Alibaba · mostly Q4_K_M No generation data · with speculation 80 t/s (1) | 1 | no generation data | no PP data | 80 1 run |
| 63 | Qwen3.6 Opus distill, abliterated35B · 3B active Alibaba · mostly i1-IQ3_XXS No generation data | 1 | no generation data | no PP data | none |
| 64 | Qwen3.8 Bonsai 227B Alibaba · mostly PQ2_0 No generation data · PP 835 t/s (1 run) | 1 | no generation data | 835 1 run | none |
| 65 | Qwen3.8 Qwen3.8-27B27B Alibaba · mostly Q4_K_S No generation data | 1 | no generation data | no PP data | none |
| 66 | Qwen3.8 Uncensored-HauHauCS27B Alibaba · mostly INT8 No generation data | 1 | no generation data | no PP data | none |
| 67 | Qwen3.8 Defiance-Uncensored27B Alibaba · mostly IQ4_XS No generation data | 1 | no generation data | no PP data | none |
| 68 | Qwen3.8 pi27B Alibaba · mostly Q4_K_M No generation data | 1 | no generation data | no PP data | none |
| 69 | Qwen3.8 Swift Uncensored27B Alibaba · mostly IQ4_XS-M No generation data | 1 | no generation data | no PP data | none |
| 70 | Qwen3.8 Concept27B Alibaba · mostly W4A16 AutoRound No generation data | 1 | no generation data | no PP data | none |
| 71 | Qwen3.8 Uncensored-HauhauCS-Aggressive27B Alibaba · mostly IQ4_XS No generation data · with speculation 72 t/s (1) | 1 | no generation data | no PP data | 72 1 run |
| 72 | Qwen3.8 Huihui-Abliterated27B Alibaba · mostly Q4_K_S No generation data | 1 | no generation data | no PP data | none |
| 73 | Qwen3.8 Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-NEO27B Alibaba · mostly Q8_0 No generation data | 1 | no generation data | no PP data | none |
| 74 | Qwen3.8 Swift-1.5-HyperQwen27B Alibaba · mostly W4A16 No generation data | 1 | no generation data | no PP data | none |
| 75 | Qwen3.8 CODER27B Alibaba · mostly IQ4_XS No generation data | 1 | no generation data | no PP data | none |
| 76 | Qwen3.8 Swift27B Alibaba · mostly IQ4_XS No generation data | 1 | no generation data | no PP data | none |
| 77 | Qwen3.8 heretic-ara27B Alibaba · mostly Q5_K_M No generation data | 1 | no generation data | no PP data | none |
| 78 | Qwen3.8 Hui Hui27B Alibaba · mostly Q4 No generation data | 1 | no generation data | no PP data | none |
| 79 | Qwen3.8 Swift-1.5-125B-A6B125B · 6B active Alibaba · mostly IQ3_S No generation data | 1 | no generation data | no PP data | none |
| 80 | Qwen3.8 Swift-1.5 Flash-Next125B · 6B active Alibaba · mostly IQ2_XS No generation data | 1 | no generation data | no PP data | none |
| 81 | gpt-oss21B · 3.6B active OpenAI No generation data | 1 | no generation data | no PP data | none |
Frequently asked
What do people run on 24GB of VRAM?
On setups with more than 16 GB and up to 24 GB of memory, the model people report most is Qwen3.8 27B, followed by Qwen3.8 125B · 6B active and Qwen3.6 27B. The table lists each with the quant most people used and the typical speed.
Does a Mac with 24GB count as 24GB of VRAM?
No. macOS keeps part of a Mac's unified memory back from the GPU, so a Mac with 24GB can use less than that for a model. The calculator's Mac tab grades every Mac against its real limit.