llamaperf

Local model performance on your hardware

Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.

What runs on your hardware?

Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Looking for a particular model?

Search for a model to see its reported speeds across GPUs and Macs.

Local model performance reports from the community

These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →

GPU: NVIDIA RTX Pro 6000 Blackwell
Compare setup details (1 active)

Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.

Tone: positiveNVIDIA hardware
generation:
150-250 tokens/s
quant:
NVFP4 (NVFP4)
kv:
fp8_e4m3
long-context

User reports running Nemotron 3.5 Super VL 120B-A12B in NVFP4 on a single RTX PRO 6000 with SGLang, getting 150 to 250 t/s generation. Setup uses a 3M token KV cache pool at fp8_e4m3 with a 786432 context length, allowing 4 concurrent requests of about 750k tokens each, with EAGLE speculative decoding (5 steps, 6 draft tokens). The user notes the model uses no system memory lookup table and is slightly faster than before; CUDA graphs work for decode but not prefill.

Oct 20, 2026
Tone: positiveNVIDIA hardware
generation:
164.8 tokens/s
quant:
W4A16-NVFP4-K32-Experts-FP8-WO (safetensors)
kv:
FP8 E4M3
agenticcodingtool-uselong-context

User reports GLM-5.3-Flash at 164.8 tok/s single-stream decode (388.2 tok/s aggregate at four concurrent requests) on two RTX PRO 6000 Blackwell GPUs. Setup is SGLang v0.4.3 SM120 with the ormandj W4A16-NVFP4-K32-Experts-FP8-WO checkpoint, FP8 E4M3 KV cache, 524,288-token context, adaptive EAGLE 5/1/6 speculative decoding and tensor parallel size 2. Cold prompt processing reaches 5,174 tok/s at 2K and 5,961 tok/s at 32K. Four simultaneous requests is the tuned operating point; at eight, aggregate falls 7.7% and median time to first token rises to 1,428 ms. The 4- and 8-GPU capacity figures are arithmetic projections, not measurements.

Oct 12, 2026
NVIDIA hardware
generation:
88.5 tokens/s
quant:
NVFP4 (safetensors)
kv:
BF16
tool-usevisionlong-contextagentic

User reports MiniMax-M3-uncensored-NVFP4 at about 88.5 tokens/s for a warmed-up single-request decode on 4x NVIDIA RTX PRO 6000 Blackwell 96 GB. Setup is SGLang with ModelOpt NVFP4 routed-expert weights, BF16 KV cache, tensor parallel size 4, 32,768-token configured context, CUDA graph enabled, and two concurrent requests allowed. The figure is a measured short-request decode rate after warmup; the checkpoint was compressed from 854.2 GB BF16 to 260.4 GB. Long-context probe recalled a sentinel from a 20,212-token prompt, and vLLM was not accepted for this deployment.

Oct 9, 2026
Tone: mixedNVIDIA hardware
generation:
133.0 tokens/s
quant:
FP8 (FP8)
kv:
fp8_e5m2

User reports GLM-4.5-Air FP8 at 133 tokens/sec with SGLang on an RTX 6000 Pro Blackwell, compared with about 78 tokens/sec in vLLM. Setup is SGLang with FP8 weights, fp8_e5m2 KV cache, 128000 context length and tensor parallelism across 4 GPUs. The user asks how to reach the same 133 tokens/sec in vLLM with Triton, and whether the Cutlass design is slower than Triton.

Oct 8, 2026
NVIDIA hardware
generation:
200.9 tokens/s
prompt processing (prefill):
1943.0 tokens/s
quant:
FP4 (FP8)
long-contexttool-usevision

User reports DeepSeek-V4.1-Flash at 200.9 t/s decode and 1,943.0 t/s prefill on 4× RTX PRO 6000 Blackwell 96GB GPUs. Setup is SGLang with FP4 routed experts and FP8 components, DSpark block 5 speculative decoding, 524,288 context, 64 GiB DDR5 cache, NVMe offload, 8 slots, GPUs capped at 275 W over PCIe without NVLink. The 200.9 t/s is the C1 single-stream figure from a 45-case sweep; total decode across 8 concurrent streams reached 713.5 t/s. The 4M populated-KV gate passed with 4,063,744 tokens across eight 500k inputs. Long-context answer quality and a one-hour soak remain unqualified.

Oct 6, 2026
Tone: positiveNVIDIA hardware
generation:
210.0 tokens/s
quant:
NVFP4 (NVFP4)
agentictool-uselong-contextcodingvision

User benchmarks Qwen3.8-27B NVFP4 at 210 tok/s single-stream on one RTX PRO 6000 Blackwell, using SGLang with the DFlash2 drafter. Setup is SGLang v0.5.20 with a 262,144-token window and 4 slots; the same model without a drafter ran 75 tok/s, and vLLM with DFlash2 ran 160 tok/s. The 27B matrix ran at a 400W power cap. User also reports 607 tok/s aggregate across 4 users, 97s full-window prefill, 73.3% BFCL core, 700/1000 arena score, and 24/40 CAPTCHA puzzles. Flash-Next and an uncensored 27B fine-tune were tested alongside.

Oct 3, 2026
NVIDIA hardware
generation:
63.7 tokens/s
quant:
NVFP4 (NVFP4)
kv:
FP8

User reports GLM-5.3-Flash at 63.7 tok/s single-request decode on 4x RTX PRO 6000 Blackwell GPUs. Setup is an experimental pinned SGLang build with an NVFP4 checkpoint, FP8 E4M3 KV cache, FlashInfer sparse MLA, and 262,144-token context, using tensor parallel TP4/EP4. With EAGLE speculative decoding (adaptive, up to 5 steps) the single-request figure rises to 82.8 tok/s. Eight concurrent requests give 183.8 tok/s aggregate baseline and 182.1 tok/s with EAGLE.

Sep 30, 2026
Tone: positiveNVIDIA hardware
generation:
77.6 tokens/s
prompt processing (prefill):
14-37 tokens/s
quant:
BF16 (safetensors)
kv:
FP8
agentictool-uselong-context

User reports Qwen3.8-27B at 77-80 tok/s on a single RTX PRO 6000 Blackwell 96GB. Setup is SGLang with BF16 safetensors, FP8 KV cache, EAGLE speculative decoding, and 262144 context length. The decode figure is a range; EAGLE accept rate was 0.94-1.00 and the user claims about 2.2x speedup over non-speculative decoding. Prefill was 14-37 tok/s on 122-token batches.

Sep 26, 2026
Tone: positiveNVIDIA hardware
prompt processing (prefill):
11352.0 tokens/s
quant:
NVFP4 (NVFP4)
tool-uselong-contextagenticcreative-writing

User reports Qwen3.8-Flash-Next on one RTX PRO 6000 Blackwell 96GB, with the official SGLang NVFP4 image cutting time to first token at roughly 254K context from 34.9s to 22.4s and raising prefill from 7,284 to 11,352 tok/s. Setup is the lmsysorg/sglang:dev-qwen38-next-local image with the RadixArk NVFP4 checkpoint, a 262,144-token window and one slot; the previous build streamed the 47.68 GiB per-layer embedding table from NVMe while the official image pins it in host RAM. A separate cache test dropped first-token time from 21.7s cold to 0.44s cached. Long-context recall scored 81/81, BFCL single-turn tool accuracy 85.2% versus 49.5% multi-turn, and tau2-bench telecom completion 68.1% with 41.2% passing all three attempts.

Sep 18, 2026

Qwen3.8 27B

2× NVIDIA RTX Pro 6000 Blackwell · SGLang · 262,144 ctx

Tone: mixedNVIDIA hardware
generation:
150.0 tokens/s
quant:
FP8
kv:
FP8
agenticlong-context

User reports Qwen 3.8 27B FP8 on two RTX 6000 Pro GPUs with sglang at 150 tk/sec, but tasks take 12x longer than Claude Opus 5. BF16 on the primary card with FP8 offloading took 3 hours versus 1h45m for FP8. Context lengths tested are 256k, 128k (too small) and 500k (faster despite being unsupported). A skill that reads large documentation files hits 80k context before starting. The user compares against Claude Code, Pi Code, Qwen Code and Hermes, and uses ChatGPT 5.6 high for testing.

Sep 7, 2026
Showing 1–10 of 10
Page 1 of 1

Community benchmarks snapshot

Records by GPU

AMD Strix Halo 128GB348NVIDIA RTX 3090271NVIDIA RTX 5090197NVIDIA DGX Spark99NVIDIA RTX 3060 12GB96NVIDIA RTX 5060 Ti 16GB88NVIDIA RTX Pro 6000 Blackwell81AMD Radeon AI PRO R9700 32GB81NVIDIA RTX 5070 Ti77NVIDIA RTX 409068

Records by model

3266 total
Qwen3.81567
Qwen3.6334
Gemma 4185
DeepSeek V4 Flash150
Qwen3.5145
DeepSeek V4.1 Flash103
other782

Records by engine

2308 total
llama.cpp1134
Strata234
vLLM227
MLX107
Ollama99
other507

Use cases

coding 937agentic 640long-context 442tool-use 281vision 174summarization 81math 77creative-writing 58multilingual 35rp 22text-generation 9reasoning 3
coding937agentic640long-context442tool-use281vision174summarization81math77creative-writing58

Median generation t/s by GPU

On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking

RTX 509080RTX Pro 600067RTX 5070 Ti55M5 Ultra 256GB50RTX 409040RX 7900 XTX40RTX 309038RX 7800 XT 16GB35RTX 5090 Laptop 24GB34RX 7900 XT32

Prompt processing, same model

M5 Ultra 256GB1800RTX 40901949RX 7900 XTX640RTX 3090561RX 7900 XT800M3 Ultra 512GB372V100 32GB941Radeon AI PRO R9700 32GB1287RTX 3080 20GB997Arc Pro B70512

Reports by model size

Qwen3.8 27B819Qwen3.8 125B · 6B active676Qwen3.6 35B · 3B active242DeepSeek V4 Flash 284B · 13B active133DeepSeek V4.1 Flash 552B · 16B active92Qwen3.6 27B81Gemma 4 26B · 4B active61GLM-5.3 320B · 18B active57

Quants

Q4_K_M395UD-Q4_K_XL189NVFP4174Q8_01194-bit106IQ4_XS86IQ3_XXS85IQ3_S84Q4_K_XL83Q480