Best GPUs for 70B and larger local LLMs
Large models need a lot of memory, so this page covers cards and Macs with 32 GB or more. Few people run dense 70B models now, and most large models people report are mixture-of-experts models that read only part of their weights per token. The ranking uses whichever large model the most of this hardware has been measured on.
What is the best GPU for 70B and larger models?
Measured on Qwen3.8 125B · 6B active at 4-bit, the fastest card people have reported is NVIDIA RTX Pro 6000 Blackwell, at 105.5 tokens per second (median of 6 runs). M4 Max 128GB follows at 57.2 tokens per second (median of 2 runs), then M5 Max 128GB at 55.0 tokens per second (one run). The table lists every card with a run of this model.
- 33-96 GBNVIDIA RTX Pro 6000 BlackwellGen 105.5 t/s
- Over 96 GBM4 Max 128GBGen 57.2 t/s
Speed only matters if the model fits. Check what fits on your card.
Ranked on Qwen3.8 125B · 6B active at 4-bit
Median tokens per second from plain runs: one GPU, one request, no speculative decoding, and the whole model in its memory. A run that kept part of the model in system RAM or on disk isn't ranked, because then the CPU and its memory set the pace as much as the GPU. The runs still differ in engine and context, and a long context can cut decode speed sharply, so each row shows what its median came from. Read a small gap between two rows as noise. Every Qwen3.8 report
| # | GPU | Memory | Gen t/s | PP t/s | Gen runs | Engine and context |
|---|---|---|---|---|---|---|
| 1 | NVIDIA RTX Pro 6000 Blackwell | 96 GB | 105.5 | 5110.06 runs | 6 | llama.cpp, ik_llama.cpp +2, 3 not stated512 to 256K context, 1 not stated |
| 2 | M4 Max 128GB | 128 GBunified | 57.2 | 586.01 run | 2 | oMLX, 1 not statedcontext not stated |
| 3 | M5 Max 128GB | 128 GBunified | 55.0 | - | 1 | engine not statedcontext not stated |
| 4 | NVIDIA Jetson AGX Thor 128GB | 128 GBunified | 36.7 | 899.01 run | 1 | engine not stated256K context |
| 5 | NVIDIA DGX Spark | 128 GBunified | 29.8 | - | 2 | llama.cpp, SGLang128K to 256K context |
| 6 | M3 Ultra 512GB | 512 GBunified | 28.0 | 558.81 run | 2 | llama.cpp, oMLX250K context, 1 not stated |
| 7 | M3 Ultra 256GB | 256 GBunified | 25.7 | - | 1 | engine not statedcontext not stated |
| 8 | AMD Strix Halo 128GB | 128 GBunified | 24.5 | 1700.55 runs | 6 | Gufo, llama.cpp, 1 not stated128K to 160K context, 3 not stated |
No speed reports yet
These match the profile, but nobody has reported a plain run on them.
What to look for
Mixture-of-experts changes the maths
A mixture-of-experts model needs memory for all its weights but reads only the active experts for each token. So it generates much faster than a dense model of the same total size, and a card can run one partly from system RAM at a usable speed. The calculator shows both memory and speed for your card.
One big card or several small ones
A single card with enough memory avoids splitting the model, which keeps setup simple. Two or more consumer cards cost less for the same memory but add setup and some speed loss between them. This page ranks single-device runs, so a multi-GPU rig won't appear here.
Macs as large-model hosts
A Mac with lots of unified memory loads large models with no special setup. It generates more slowly than a data center card on the same model, and the table shows by how much.
Frequently asked
How much VRAM do I need for a 70B model?
It depends on the quant and the context length. Each model's VRAM page lists the memory for every size at every common quant, and the calculator checks it against your card.
Can I run a large model on a single 24 GB card?
A dense 70B only with heavy quantization or offloading, both of which cost quality or speed. A mixture-of-experts model can run with its experts in system RAM, and the calculator estimates the speed you'd get.
How is this list ranked?
Every card is ranked on its median speed for one large reference model at one quant level, counting plain single-device runs only. The reference is the model of 60B parameters or more that the most of this hardware has reports of.