Local model performance on your hardware

generation:

244.0 tokens/s

prompt processing (prefill):

5230.0 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Flint 40B-A2.5B at 244 t/s single-request decode and 5,230 t/s prefill on one RTX PRO 6000 Blackwell 96 GB, comparing it with Ornith 1.5 35B-A3B and Nemotron 3.5 Lightning 30B-A3B in an agentic coding run. Setup is llama.cpp b11803 with Q8_0 GGUF, 64K context per slot and --parallel 3, three workers running in parallel behind Opus 5.5 as planner. Flint finished in 7:36 with 6 fixes and $0.41 of Opus spend, against Ornith's 9:04, 14 fixes and $0.49. Nemotron had the best decode and prefill on paper but generated about 5x more tokens and needed six fix waves. One run per model, variance not measured; Q8_0 on CUDA, so NVFP4/MLX numbers will differ.

generation:

138.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports GLM-4.5-Air at about 138 tok/s on an RTX PRO 6000 96GB. Setup is llama.cpp with Q4_K_M, measured at empty context. The figure comes from a site of 941 measured rows, including 540 llama-bench rows the user ran on rented GPUs from an RTX 3060 12GB through an RTX 5090 and an RTX PRO 6000 96GB.

generation:

150-250 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

User reports running Nemotron 3.5 Super VL 120B-A12B in NVFP4 on a single RTX PRO 6000 with SGLang, getting 150 to 250 t/s generation. Setup uses a 3M token KV cache pool at fp8_e4m3 with a 786432 context length, allowing 4 concurrent requests of about 750k tokens each, with EAGLE speculative decoding (5 steps, 6 draft tokens). The user notes the model uses no system memory lookup table and is slightly faster than before; CUDA graphs work for decode but not prefill.

generation:

81-88 tokens/s

prompt processing (prefill):

12000.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8 Flash-Next NVFP4 at 81-88 tokens per second on a single session on one RTX Pro 6000, without MTP. Setup follows the primitive-ai NVFP4 documentation; the 51B n-gram table needs 100GB of spare system memory. Prompt processing is GPU compute bound at 12000 tokens per second even with context scaled to 100k. Throughput scales to 341 tokens per second with 6 concurrent clients. The user calls the model great and is still benchmarking with MTP.

generation:

132.5 tokens/s

quant:

BF16 (BF16)

User reports Gemma 3 4B at 132.52 t/s on an RTX 6000 Pro Blackwell, a 3.34x speedup over Ollama. Setup is vLLM (dev build) with BF16 weights and MTP speculative decoding using 3 draft tokens, single request, ~512-token prompt and 64 output tokens, averaged over 3 runs. The user also tested Qwen3-VL 8B (Abliterated) with 1 MTP layer, noting Gemma 3's 3 MTP layers explain its larger gain; going beyond the MTP layer count as draft tokens does not help.

generation:

500.0 tokens/s

User reports Qwen 3.6 35B-A3B at 500 tok/s decode on a machine with an RTX Pro 6000 and 2x RTX 3090 (96GB + 24GB + 24GB VRAM) and 128GB system RAM. Setup is NInfer; quant and context length are not stated. The 500 tok/s figure is for fast, simple tasks; the user's main agentic driver is Qwen Flash Next at Q6, with DSv4F+vision at Q8 as a tossup. Qwen 3.8 27B was the previous driver but struggles on long autonomous runs with 250+ tool uses per turn.

generation:

232.0 tokens/s

prompt processing (prefill):

4315.0 tokens/s

quant:

NVFP4 (NVFP4)

User compares Ninfer6000, vLLM and llama.cpp running Qwen3.8-Flash-Next on an RTX Pro 6000 Blackwell 96GB. Ninfer6000 reaches 232 t/s decode at ~1k prompt versus 159 t/s for vLLM NVFP4 and 138 t/s for llama.cpp Q4_K_XL, with a mean decode of 238 t/s across prompts up to 114k. Setup is Ninfer6000 with an NVFP4 conversion, vLLM with nvidia/Qwen3.8-Flash-Next-NVFP4 and llama.cpp with unsloth UD-Q4_K_XL, all on one RTX Pro 6000 Blackwell at 600W with a Ryzen 5 5600X and 96GB DDR4-3000. Prefill trails vLLM (4,315 t/s vs 6,944 t/s at ~1k) but beats llama.cpp. Accuracy benchmarks are close across all three, with Ninfer6000 ahead on BFCL v4 tool calling (76.67 vs 74.56 and 73.00). User reports a crippling tool-calling formatting issue with Ninfer6000 in real-world use in Open-WebUI and Cline, and will keep using vLLM until it is resolved.

prompt processing (prefill):

16459.6 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks a GDN chunked kernel pull request in llama.cpp, reporting prompt-processing throughput on several GPUs and models. On an RTX PRO 6000 Blackwell with Qwen3.5-35B-A3B-Q8_0 at 2048 prompt tokens, the PR reaches 16459.56 tok/s against 13653.58 tok/s on master, a 20.6% gain. The table also covers RTX 5090, RTX 4090, RTX 3090 and Spark GB10 across Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, Qwen3.8-27B and Qwen3.8-Flash-Next at 2048 and 4096 prompt tokens.

generation:

205.3 tokens/s

quant:

IQ3_S (GGUF)

User benchmarks Strata 0.1.39 against llama.cpp on Qwen3.8-Flash-Next (IQ3_S) on an RTX PRO 6000 Blackwell 96GB, reporting 205.263 tok/s generation at a 4K prompt versus 102.889 tok/s for llama.cpp. Setup is Strata with its draft head on, one request at a time, 256 tokens written greedy, median of three runs; the whole model sat on the card. Strata also led on a 3090 Ti (98.291 vs 22.490 tok/s) and a 5090 laptop (85.6 vs 24.4 tok/s), where the model did not fit and experts were kept in system RAM. Strata pins 46.84 GiB of system RAM for this file on every card.

generation:

164.8 tokens/s

quant:

W4A16-NVFP4-K32-Experts-FP8-WO (safetensors)

kv:

FP8 E4M3

User reports GLM-5.3-Flash at 164.8 tok/s single-stream decode (388.2 tok/s aggregate at four concurrent requests) on two RTX PRO 6000 Blackwell GPUs. Setup is SGLang v0.4.3 SM120 with the ormandj W4A16-NVFP4-K32-Experts-FP8-WO checkpoint, FP8 E4M3 KV cache, 524,288-token context, adaptive EAGLE 5/1/6 speculative decoding and tensor parallel size 2. Cold prompt processing reaches 5,174 tok/s at 2K and 5,961 tok/s at 32K. Four simultaneous requests is the tuned operating point; at eight, aggregate falls 7.7% and median time to first token rises to 1,428 ms. The 4- and 8-GPU capacity figures are arithmetic projections, not measurements.

generation:

86.0 tokens/s

User benchmarks Qwen3-8B on an RTX PRO 6000 Blackwell 96GB, reporting 86 tok/s single-stream decode in BF16 under vLLM. Setup is vLLM 0.27.1 with BF16 weights, greedy decoding, concurrency 32 for aggregate figures; SGLang 0.5.9 and llama.cpp CUDA builds were also tested. Aggregate throughput at concurrency 32 was 1,725 tok/s for vLLM, 1,327 for SGLang and 428 for llama.cpp; single-stream decode sat in an 83 to 96 tok/s band across all stacks. An FP8 checkpoint on vLLM reached 130 tok/s single-stream and 2,597 tok/s aggregate, with p50 latency down from 0.74 s to 0.49 s.

prompt processing (prefill):

3319.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

INT8

User reports Qwen3.8-Flash-Next UD-Q4_K_XL on an RTX PRO 6000 with Strata, INT8 KV cache at 256K context, and MTP-4 speculative decoding. Single-request decode varies by task: 185.8 t/s prose, 320.4 t/s counting, 298.9 t/s coding, 289.1 t/s reasoning, with prefill at 3,319 tok/s for a 17.3K prompt. vLLM on the same GPU with NVFP4 quant reached 95.3-171.0 t/s decode and 4,084 tok/s prefill, while vLLM on DGX Spark TP2 reached 38.2-75.3 t/s decode and 2,825 tok/s prefill.

generation:

69.9 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

Q8_0

User reports GLM-5.3-Flash at 69.89 t/s on an RTX PRO 6000 Blackwell 96 GB. Setup is llama.cpp with an IQ2_XXS GGUF and Q8_0 KV cache, fully resident on the card. Decode falls to 52.74 t/s at 8192 context, 31.70 at 32768 and 12.24 at 131072. The same file on a single RTX 3090 24 GB with 64 GB DDR5 and -ncmoe 40 gives 11.55 t/s decode and 36.69 t/s prefill. The IQ3_XXS build is not benchmarked.

generation:

113.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

MXFP4 (MXFP4)

kv:

BF16

User reports MiniMax-M3 at ~113 tok/s single-stream decode on 4x RTX PRO 6000 Blackwell (96 GB each). Setup is vLLM with the MXFP4 checkpoint (~256 GB, ~60 GB per card), BF16 KV cache, tensor parallelism 4, and 250K context; cold prefill runs ~2,800 tok/s flat from 8K to 131K. Four concurrent requests reach ~184 tok/s aggregate. A second NVFP4/SGLang configuration on the same host decodes at ~87 tok/s single-stream with a ~643K-token KV pool and image/video support.

generation:

88.5 tokens/s

quant:

NVFP4 (safetensors)

kv:

BF16

User reports MiniMax-M3-uncensored-NVFP4 at about 88.5 tokens/s for a warmed-up single-request decode on 4x NVIDIA RTX PRO 6000 Blackwell 96 GB. Setup is SGLang with ModelOpt NVFP4 routed-expert weights, BF16 KV cache, tensor parallel size 4, 32,768-token configured context, CUDA graph enabled, and two concurrent requests allowed. The figure is a measured short-request decode rate after warmup; the checkpoint was compressed from 854.2 GB BF16 to 260.4 GB. Long-context probe recalled a sentinel from a 20,212-token prompt, and vLLM was not accepted for this deployment.

generation:

45.9 tokens/s

prompt processing (prefill):

362.3 tokens/s

quant:

q2-imatrix (GGUF)

User reports DeepSeek-V4-Flash q2-imatrix at 45.9 t/s generation and 362.3 t/s prefill on an RTX Pro 6000 Blackwell 96GB at 2048 context. Setup is ds4 with q2-imatrix GGUF, fp16 dequant cache disabled and MTP off, fully resident at ~91 GB peak. The 80.76 GB model only fits with DS4_CUDA_Q8_F16_CACHE_MB=0; MTP adds 3.5 GB and pushes over 96 GB. Generation beats posted M5 Max and DGX Spark GB10 q2 numbers; prefill is about equal to Spark. At 28672 context generation drops to 40.6 t/s. The user notes the card is tight for this model and heavier ds4_test sessions OOM.

generation:

110.0 tokens/s

quant:

Q5 (GGUF)

User reports Qwen3.8 Flash-Next Abliterated at 110 tok/s on an RTX Pro 6000 Blackwell 96GB. Setup uses Q5 or Q6 quantization with 128k context. The user rents the GPU monthly and only runs it during the day, leaving it idle for about 8 hours each night.

generation:

133.0 tokens/s

quant:

FP8 (FP8)

kv:

fp8_e5m2

User reports GLM-4.5-Air FP8 at 133 tokens/sec with SGLang on an RTX 6000 Pro Blackwell, compared with about 78 tokens/sec in vLLM. Setup is SGLang with FP8 weights, fp8_e5m2 KV cache, 128000 context length and tensor parallelism across 4 GPUs. The user asks how to reach the same 133 tokens/sec in vLLM with Triton, and whether the Cutlass design is slower than Triton.

generation:

36.1 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

MXFP4

User reports DeepSeek V4 Flash MXFP4 at 36.1 t/s decode and 497 t/s prefill served across a Radeon 8060S iGPU and an RTX PRO 6000 Blackwell. Setup is llama.cpp with the rocm-cuda backend on two GPUs, the Strix Halo iGPU plus an external RTX PRO 6000 96 GB, MXFP4 weights (146 GB) fully GPU-resident with no CPU offload and the experts of 18 layers on the iGPU, under Windows. The repo also reports that on Blackwell a CUDA 13.3 build of llama.cpp loses 50-80% of decode past 8192 context: an unnamed 27B model on the RTX PRO 6000 drops from 72 to 35 t/s, while a CUDA 12.8 build of the same commit keeps 74.6 t/s at 8K. gpt-oss-120b runs 52 t/s on the iGPU alone, and Qwen3.8-Flash-Next Q4_K_M across two AMD GPUs gave 21.5-25.8 t/s decode at 4K.

generation:

90.6 tokens/s

quant:

NVFP4 (NVFP4)

kv:

BF16

User reports GLM-5.1 at 90.6 t/s single-stream decode on 8x RTX PRO 6000 Blackwell (96 GB each, 768 GB total). Setup is vLLM with an NVFP4 checkpoint (lukealonso/GLM-5.1-NVFP4-MTP), BF16 KV cache, DCP1, MTP speculative decoding on, 371456 KV cache tokens, max model length 202752. Aggregate decode reaches 1165.1 t/s at 64 concurrent requests; cold prefill is 4333 t/s at 8k context and 3170 t/s at 128k. DCP2 and DCP4 trade decode speed for larger KV cache, and DCP8 MTP-off was unstable at long context.