NVIDIA RTX Pro 6000 Blackwell reports

prompt processing (prefill):

11352.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on one RTX PRO 6000 Blackwell 96GB, with the official SGLang NVFP4 image cutting time to first token at roughly 254K context from 34.9s to 22.4s and raising prefill from 7,284 to 11,352 tok/s. Setup is the lmsysorg/sglang:dev-qwen38-next-local image with the RadixArk NVFP4 checkpoint, a 262,144-token window and one slot; the previous build streamed the 47.68 GiB per-layer embedding table from NVMe while the official image pins it in host RAM. A separate cache test dropped first-token time from 21.7s cold to 0.44s cached. Long-context recall scored 81/81, BFCL single-turn tool accuracy 85.2% versus 49.5% multi-turn, and tau2-bench telecom completion 68.1% with 41.2% passing all three attempts.

generation:

150.0 tokens/s

prompt processing (prefill):

10000.0 tokens/s

quant:

8 bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 8-bit on 2 RTX Pro 6000 Blackwell GPUs at 150 t/s decode and 10K t/s prefill. Setup uses peer-to-peer communication, CUDA graphs, tensor parallelism, and both GPUs capped at 500W. The user also runs DeepSeek V4 Flash 0731 with 1M context and room for 4 concurrent requests. User finds the system fast and plans to cancel subscriptions; asks about excluding thinking tokens from Qwen context and other models to try.

quant:

GPU-5 (GGUF)

User reports ShapeLearn GGUFs for Qwen 3.8 27B, with the 3.84 bpw (GPU-5) quant reaching 99.63% of BF16's aggregate score across 8 benchmarks and the 3.23 bpw (GPU-4) quant reaching 98.72%. Benchmarks were run on RTX Pro 6000 Blackwell, RTX 5090, RTX 4090, RTX 3090, RTX 4080 and RTX 5060 Ti. DFlash2 delivered 1.34-2.10x baseline throughput and MTP delivered 1.28-1.66x with temperature sampling. User notes that Unsloth Dynamic V3's UD-IQ3_S had ~20% lower KLD than a similarly sized Lite model but scored 95.55% versus Lite's 97.33% of BF16's aggregate benchmark score.

generation:

101.9 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares llama.cpp, SGLang, and FreeToken on Qwen3.8-Flash-Next. At full context, TTFT is 35.4s for SGLang, 80.4s for FreeToken, 210.2s for llama.cpp with MTP, and 258.4s for llama.cpp baseline. Decode at full context is 126.9 t/s for SGLang, 87.5 t/s for FreeToken, 52.6 t/s for llama.cpp with MTP, and 20.3 t/s for llama.cpp baseline. MTP improves decode 1.63x at 8K and 1.69x at 32K. Accuracy is 95.22-95.75% on GSM8K and 92.20-93.00% on MATH-500. Startup takes 16s for llama.cpp, 108s for SGLang, and 126s for FreeToken.

generation:

57.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Glimmer 30B at ~57 t/s with speculative decoding using DFlash, against ~25 t/s without speculation, a 2.3x speedup. Setup is vLLM with six patches applied to the image. Mean acceptance length is ~2.5 tokens per verification step, with overall draft acceptance ~10%. Per-position acceptance is ~73% at position 0, ~40% at 1, ~15% at 2, and near zero past 5. The recipe's 3.1x figure was greedy decoding with K-quant on llama.cpp under different conditions. The predicted ceiling is ~65 t/s at 2.6 mean acceptance.

generation:

35.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks TensorSharp against llama.cpp on an RTX PRO 6000 Blackwell, with plain text generation decode at about 35 tok/s at 60 prompt tokens. The user also tests DFlash speculative decoding and 2x RTX PRO 4000 Blackwell 24GB with tensor parallelism.

generation:

128.6 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 100-130 t/s on a single RTX 6000 Pro. Setup is vLLM-Moet at 130K context, requiring about 150GB RAM for loading.

generation:

27.2 tokens/s

prompt processing (prefill):

14860.0 tokens/s

quant:

Q3_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 27.2 t/s on an RTX 6000 96GB and a W7800 48GB. Setup is llama.cpp server with the Q3_K_XL quant. User notes it is cheaper than K3 and GLM 5.2 but very good.

generation:

29.0 tokens/s

prompt processing (prefill):

1328.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 (Q4) at ~29 tok/s decode on one RTX PRO 6000 96GB. Setup is Krasis v1.0.19 with INT4 experts, keeping the hottest 6,440 of 11,008 routed experts in VRAM and serving the rest from system RAM during decode. Decode after a roughly 1K prompt was 29.4, 28.2 and 28.5 tok/s for 50, 100 and 250 tokens, and 19.4 tok/s after a 62K prompt. Prefill reaches 1,328 tok/s at 23,348 tokens and 1,204 tok/s at 62,403, in internal timing-disabled results rather than HTTP round trips.

quant:

BF16

rating:

4/5

User reports Qwen 3.6 27B abliterated BF16 running locally via vLLM with llama-swap and MTP speculative decoding, scoring 8/10 on a Terminal-Bench 2.0 pilot. The same setup beats DeepSeek-V4 IQ2 at 7/10 and comes close to an FP8 API at 9/10. It is the only configuration to pass the cancel-async-tasks hard task, and it missed build-cython-ext and sqlite-db-truncate on timeout.

generation:

45.0 tokens/s

quant:

Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731-UD-Q4_K_XL at 45 t/s and 512k context as their primary coding model, across a multi-GPU setup of an RTX 6000 Pro Blackwell 96GB, 2x RTX 5090, an RTX 4090, 3x AMD R9700, and a Strix Halo laptop. Other models in use are GLM-4.7-flash, Gemma-4-12B-it, KAT-Coder-V2.5-Dev, LFM2.5-VL-1.6B, GLM-5.2-UD, Kimi-K2.7-Code, and MiniMax-M3. The user is dissatisfied with the workflow, describing it as a good fast worker or a good thinker but not both, and notes that switching models kills the cache.

generation:

31.2 tokens/s

quant:

Q4_K_XL

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DSpark speculative decoding with the drafter in RAM running faster than in VRAM. Setup uses a q8_0 KV cache, which allows 768K context with minimal decode loss. LiveCodeBench scores 28/30 (93.3%).

generation:

114.8 tokens/s

quant:

FP8

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-27B-FP8 against Qwen3.6-27B-FP8 on an RTX PRO 6000 Blackwell with vLLM. In an MTP sweep, Qwen3.8 peaks at MTP 6 with 114.8 tok/s and Qwen3.6 peaks at MTP 7 with 131.6 tok/s. Qwen3.8 is 5-20% slower across MTP steps, though quality is comparable or better at some steps. A pre-sweep manual run at MTP 5 showed Qwen3.8 faster, at 108.9 vs 103.6 tok/s.

generation:

99.6 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports that tuning the MTP draft window from n-max 12 to 5 improved throughput from 63 to 99.58 t/s. The Q4_K_XL quant reached 125.63 t/s, faster than Q8_0 at 99.58 t/s, with no quality gap detected. The user recommends reasoning_effort medium over xhigh or low, and notes that model quality revealed harness bugs.

generation:

56.0 tokens/s

prompt processing (prefill):

3000.0 tokens/s

quant:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 56 t/s decode and about 3K t/s prefill on a single RTX PRO 6000 96GB GPU. Setup is vLLM with an 8-bit quant at about 262K context, serving three concurrent users. The figures are for three concurrent users, and the user offers free community access rather than presenting a benchmark.

generation:

104.8 tokens/s

prompt processing (prefill):

12400.0 tokens/s

quant:

groupwise-int

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B on an RTX PRO 6000 Blackwell at 104.83 tok/s weighted decode, running a long-horizon agentic task at 262K context. Setup is NInfer with a groupwise-int quant (mixed Q4/Q5/Q6). The run covered 966 model calls, 131.2M input tokens and 853.3K output tokens. Prompt processing was about 12.4K tok/s, measured client-side and including queue time. The user plans to try an NVFP4 profile and estimates API cost savings of $650+.

generation:

150.0 tokens/s

quant:

FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B FP8 on two RTX 6000 Pro GPUs with sglang at 150 tk/sec, but tasks take 12x longer than Claude Opus 5. BF16 on the primary card with FP8 offloading took 3 hours versus 1h45m for FP8. Context lengths tested are 256k, 128k (too small) and 500k (faster despite being unsupported). A skill that reads large documentation files hits 80k context before starting. The user compares against Claude Code, Pi Code, Qwen Code and Hermes, and uses ChatGPT 5.6 high for testing.

quant:

Q8

User benchmarks Qwen3.6-27B at 32 concurrent clients on the Paddock engine, with TTFT of 697 ms against 2.5 s for vLLM and 6.9 s for llama.cpp. User also reports Qwen3.8-27B at Q8 on an RTX PRO 6000 with speculation on and off: 46.8 to 202 tok/s single stream, 320 to 822 at eight concurrent chats, and 1005 to 1285 at 32. The engine is free but not open source.

generation:

153.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks DFlash 2 speculative decoding on Qwen 3.8 27B with a llama.cpp PR build on an RTX PRO 6000 Blackwell 96GB and a Ryzen 9 9950X. Setup is llama.cpp with 262144 context, an f16 KV cache and full offload. The user reports 2.26x speedup on LiveCodeBench, from 67.97 to 153.91 tok/s, and 4.68x on multi-turn coding with n-gram lookup, and gives a detailed analysis of n-gram drafters and draft width.

generation:

114.2 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen 3.8 27B community quants on an RTX Pro 6000 96GB, with atomic ad-q6_k averaging 114.17 t/s as the best. The quants tested are atomic ad-q6_k, unsloth ud-q6_k_l and bartowski q6_k. The user also compares the results against a Claude Opus 4.6 subscription, with prompts generating self-playing 3D HTML games.