Local model performance on your hardware

generation:

69.9 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 69.89 t/s on an RTX PRO 6000 Blackwell 96 GB. Setup is llama.cpp with an IQ2_XXS GGUF and Q8_0 KV cache, fully resident on the card. Decode falls to 52.74 t/s at 8192 context, 31.70 at 32768 and 12.24 at 131072. The same file on a single RTX 3090 24 GB with 64 GB DDR5 and -ncmoe 40 gives 11.55 t/s decode and 36.69 t/s prefill. The IQ3_XXS build is not benchmarked.

generation:

36.1 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash MXFP4 at 36.1 t/s decode and 497 t/s prefill served across a Radeon 8060S iGPU and an RTX PRO 6000 Blackwell. Setup is llama.cpp with the rocm-cuda backend on two GPUs, the Strix Halo iGPU plus an external RTX PRO 6000 96 GB, MXFP4 weights (146 GB) fully GPU-resident with no CPU offload and the experts of 18 layers on the iGPU, under Windows. The repo also reports that on Blackwell a CUDA 13.3 build of llama.cpp loses 50-80% of decode past 8192 context: an unnamed 27B model on the RTX PRO 6000 drops from 72 to 35 t/s, while a CUDA 12.8 build of the same commit keeps 74.6 t/s at 8K. gpt-oss-120b runs 52 t/s on the iGPU alone, and Qwen3.8-Flash-Next Q4_K_M across two AMD GPUs gave 21.5-25.8 t/s decode at 4K.

prompt processing (prefill):

13816.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a prompt-processing regression on Qwen3.6-35B-A3B, dropping from 13816.18 tokens per second to 6858.24 tokens per second between llama.cpp builds 4ebdf2c74 and bed0a8566. Setup is llama.cpp with Q4_K_M GGUF, 32768 context, -ngl 99 across 3x RTX Pro 6000 Blackwell (96 GB each) on a Threadripper PRO 9975WX with 256 GB RAM. The regression appears in llama-completion and llama-server but not in llama-bench, where both builds measure about 15600 t/s at pp18606.

generation:

118.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B with MTP speculative decoding at 117.95 tok/s on an RTX Pro 6000 Blackwell, compared to 43.22 tok/s without MTP, a 2.73x speedup. Setup is llama.cpp with Q8_0 GGUF weights, 3000 token context, and 4 speculative tokens. Hardware is an RTX Pro 6000 Blackwell with 96GB VRAM and 92GB system RAM. A separate vLLM run of Gemma 4 31B FP8 with MTP averaged 124.26 tok/s over 10 runs, totaling 4740 tokens in 38.14s.

generation:

101.9 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares llama.cpp, SGLang, and FreeToken on Qwen3.8-Flash-Next. At full context, TTFT is 35.4s for SGLang, 80.4s for FreeToken, 210.2s for llama.cpp with MTP, and 258.4s for llama.cpp baseline. Decode at full context is 126.9 t/s for SGLang, 87.5 t/s for FreeToken, 52.6 t/s for llama.cpp with MTP, and 20.3 t/s for llama.cpp baseline. MTP improves decode 1.63x at 8K and 1.69x at 32K. Accuracy is 95.22-95.75% on GSM8K and 92.20-93.00% on MATH-500. Startup takes 16s for llama.cpp, 108s for SGLang, and 126s for FreeToken.

generation:

27.2 tokens/s

prompt processing (prefill):

14860.0 tokens/s

quant:

Q3_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 27.2 t/s on an RTX 6000 96GB and a W7800 48GB. Setup is llama.cpp server with the Q3_K_XL quant. User notes it is cheaper than K3 and GLM 5.2 but very good.

generation:

31.2 tokens/s

quant:

Q4_K_XL

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DSpark speculative decoding with the drafter in RAM running faster than in VRAM. Setup uses a q8_0 KV cache, which allows 768K context with minimal decode loss. LiveCodeBench scores 28/30 (93.3%).

generation:

99.6 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports that tuning the MTP draft window from n-max 12 to 5 improved throughput from 63 to 99.58 t/s. The Q4_K_XL quant reached 125.63 t/s, faster than Q8_0 at 99.58 t/s, with no quality gap detected. The user recommends reasoning_effort medium over xhigh or low, and notes that model quality revealed harness bugs.

generation:

153.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks DFlash 2 speculative decoding on Qwen 3.8 27B with a llama.cpp PR build on an RTX PRO 6000 Blackwell 96GB and a Ryzen 9 9950X. Setup is llama.cpp with 262144 context, an f16 KV cache and full offload. The user reports 2.26x speedup on LiveCodeBench, from 67.97 to 153.91 tok/s, and 4.68x on multi-turn coding with n-gram lookup, and gives a detailed analysis of n-gram drafters and draft width.

generation:

109.1 tokens/s

prompt processing (prefill):

1955.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 Flash Next across VRAM tiers from 8 GB to 96 GB with llama.cpp b10666. CPU-only decode reaches 8.34 tok/s and full 96 GB decode reaches 109.07 tok/s. At 245K context, decode ranges from 14.89 tok/s on 24 GB to 21.61 tok/s on 96 GB. The model activates 6B params per token (MoE). RAM-resident loading gives 1.87x prefill versus mmap. Non-unified KV reaches 92.0 tok/s total at concurrency 16. A PLE table on CUDA causes a 55.6x decode slowdown, from 108.5 to 1.95 tok/s.

generation:

57.4 tokens/s

quant:

Q6_K (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Tencent's Hy3, a 295B MoE with 21B active, at 57.4 tok/s generation with the Q6_K GGUF on 8x RTX PRO 6000 Blackwell. Setup is llama.cpp llama-bench with layer split and F16 KV cache, with the MTP/NextN block excluded. On the same cards Q4_K_M runs at 67.3 tok/s, Q3_K_L at 63.3 tok/s and IQ2_M at 78.7 tok/s. The user's Nemotron-Labs-Audex-30B-A3B quants run at 287 (Q8_0), 334 (Q5_K_M), 345 (Q4_K_M) and 329 (MXFP4_MOE) tok/s on 2x RTX PRO 6000 Max-Q.