llamaperf

Best local LLMs for 96GB VRAM

How far 96GB of graphics memory goes for local models on cards like NVIDIA RTX Pro 6000 Blackwell or NVIDIA RTX PRO 6000 Max-Q: which models fit, and what people actually run on it.

Is 96GB of VRAM enough?

96GB holds a dense model of up to about 137B parameters at Q4_K_M with an 8K context, by the calculator's memory estimate. The largest model people report here that fits comfortably is Qwen3.8 125B · 6B active, which needs about 72.3 GB at Q4_K_M with an 8K context. Anything bigger needs a smaller quant, part of the model in system RAM, or more memory, and a longer context needs more room on top.

The calculator checks your exact card, including longer contexts and offloading.

Models that fit in 96GB

Models people report on llamaperf, largest first, at the best quant whose memory with an 8K context stays within the calculator's comfortable band (85% of the card). Estimated from each model's shape.

And 131 smaller models. The calculator lists them all for your card.

What people run on 96GB

Model sizes reported on setups with more than 64 GB and up to 96 GB of memory, most reported first. Typical speed is the median of plain runs, and speculative runs are counted apart.

#ModelReports
1Qwen3.8 Flash-Next125B · 6B active
Alibaba · mostly NVFP4
Gen 105 t/s (6 runs) · PP 5110 t/s (6 runs) · with speculation 90 t/s (11)
52
2Qwen3.827B
Alibaba · mostly FP8
Gen 18 t/s (3 runs) · with speculation 100 t/s (11)
30
3DeepSeek V4 Flash284B · 13B active
DeepSeek · mostly Q4_K_XL
Gen 45 t/s (4 runs) · PP 312 t/s (2 runs)
17
4GLM-5.3 Flash320B · 18B active
Zhipu AI · mostly 3
Gen 70 t/s (1 run)
9
5Qwen3.627B
Alibaba · mostly 8bit
Gen 7.4 t/s (1 run) · PP 121 t/s (1 run) · with speculation 28 t/s (3)
7
6Qwen3.635B · 3B active
Alibaba · mostly Q4_K_M
Gen 290 t/s (1 run) · PP 11486 t/s (1 run)
6
7DeepSeek V4.1 Flash552B · 16B active
mostly VQ2
No generation data
6
8Qwen3.535B · 3B active
Alibaba · mostly Q8_0
Gen 92 t/s (2 runs) · PP 1808 t/s (3 runs)
3
9GLM-4.5-Air106B · 12B active
mostly Q4_K_M
Gen 138 t/s (1 run)
2
10gpt-oss117B · 5.1B active
OpenAI · mostly MXFP4
Gen 125 t/s (2 runs) · PP 347 t/s (1 run)
2
11Gemma 431B
Google DeepMind · mostly NVFP4
Gen 51 t/s (1 run) · PP 15359 t/s (1 run) · with speculation 125 t/s (1)
2
12Qwen38B
Alibaba · mostly Q4
Gen 43 t/s (1 run)
2
13Muse Glimmer30B
Meta · mostly BF16
Gen 35 t/s (1 run) · with speculation 57 t/s (1)
2
14DeepSeek V4 Pro1600B · 49B active
DeepSeek
No generation data
2
15Qwen3.8 Swift-1.527B
Alibaba · mostly 4.7bpw
No generation data
2
16Qwen3.8 Qwen3.8-27B27B
Alibaba · mostly Q4_K_M
No generation data
2
17Qwen3.84B
Alibaba · mostly Q4_K_M
Gen 291 t/s (1 run)
1
18Qwen3.8 Flash-Next Abliterated125B · 6B active
Alibaba · mostly Q5
Gen 110 t/s (1 run)
1
19Qwen330B · 3B active
Alibaba · mostly Q4_K_M
Gen 59 t/s (1 run) · PP 920 t/s (1 run)
1
20DeepSeek V4.1 Flash763B
mostly Q8_0
No generation data
1
21GLM-4.7 Flash30B
Zhipu AI
No generation data · with speculation 168 t/s (1)
1
22GLM-5.1744B · 40B active
Zhipu AI · mostly NVFP4
No generation data
1
23Gemma 34B
Google · mostly BF16
No generation data · with speculation 133 t/s (1)
1
24Gemma 4 Gemma 4 31B31B
Google DeepMind · mostly Q8_0
No generation data
1
25Kimi K3.1 Pro-Coder2800B
mostly 2.81/2.56/1.44 bpw 3-tier MoE (~1.92 bpw avg)
No generation data
1
26Llama 3.21B
Meta
No generation data
1
27Mimo 2.5310B · 15B active
Xiaomi
No generation data
1
28Mimo 2.6 Flash-RL309B · 15B active
Xiaomi · mostly 2.20 bpw
No generation data
1
29MiniMax M3.1 Flash428B · 23B active
mostly 2-bit
No generation data
1
30Minimax M2.1230B · 10B active
mostly UD-Q4_K_XL
No generation data
1
31Minimax M3 uncensored428B · 23B active
MiniMax · mostly NVFP4
No generation data
1
32Minimax M3428B · 23B active
MiniMax · mostly MXFP4
No generation data
1
33Muse Muse Glimmer 30B28B
Meta · mostly Q8_0
No generation data
1
34Nemotron 3.5 Super VL120B · 12B active
mostly NVFP4
No generation data
1
35Qwen3-Next80B · 3B active
Alibaba · mostly 4-bit
No generation data
1
36Qwen3.5 NuExtract34B
Alibaba
No generation data
1
37Qwen3.6 abliterated27B
Alibaba · mostly BF16
No generation data
1
38Qwen3.6 Qwen3.6-35B-A3B35B · 3B active
Alibaba · mostly Q8_0
No generation data
1
39Qwen3.6 Ornith-Agents-A1-3.635B · 3B active
Alibaba · mostly Q4_K_M
No generation data
1
40Qwen3.8 sabi-noema27B
Alibaba · mostly 6bit
No generation data
1
41Qwen3.8 Swift-1.5 Flash-Next125B · 6B active
Alibaba · mostly IQ3_S
No generation data · with speculation 145 t/s (1)
1
42Step-3.5-Flash196B · 11B active
mostly IQ4_XS
No generation data
1
43Tencent-HY3295B · 21B active
Tencent · mostly Q6_K
No generation data
1

Frequently asked

What do people run on 96GB of VRAM?

On setups with more than 64 GB and up to 96 GB of memory, the model people report most is Qwen3.8 125B · 6B active, followed by Qwen3.8 27B and DeepSeek V4 Flash 284B · 13B active. The table lists each with the quant most people used and the typical speed.

Does a Mac with 96GB count as 96GB of VRAM?

No. macOS keeps part of a Mac's unified memory back from the GPU, so a Mac with 96GB can use less than that for a model. The calculator's Mac tab grades every Mac against its real limit.