NVIDIA RTX Pro 6000 Blackwell reports

generation:

172.0 tokens/s

prompt processing (prefill):

5905.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Flash Next at 172.0 tok/s decode without speculative decoding on an RTX6000 96GB, at 512 context with non-expert weights downsampled from 16-bit to 8-bit. Setup is NInfer6000, the user's fork of NInfer, with NVFP4 quants and prefill at 5,905 tok/s on 512 tokens and 13,908 tok/s on 8,192 tokens. With MTP3 and --lm-head-draft the same setup reaches 274.8 tok/s at 512 and 401.3 tok/s at 8K. Keeping non-experts at 16-bit gives 118.0 tok/s without speculative decoding.

generation:

200.9 tokens/s

prompt processing (prefill):

1943.0 tokens/s

quant:

FP4 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 200.9 t/s decode and 1,943.0 t/s prefill on 4× RTX PRO 6000 Blackwell 96GB GPUs. Setup is SGLang with FP4 routed experts and FP8 components, DSpark block 5 speculative decoding, 524,288 context, 64 GiB DDR5 cache, NVMe offload, 8 slots, GPUs capped at 275 W over PCIe without NVLink. The 200.9 t/s is the C1 single-stream figure from a 45-case sweep; total decode across 8 concurrent streams reached 713.5 t/s. The 4M populated-KV gate passed with 4,063,744 tokens across eight 500k inputs. Long-context answer quality and a one-hour soak remain unqualified.

generation:

101.6 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next (125B-A6B) on a single RTX PRO 6000 Blackwell 96GB with 64GB host RAM, comparing vLLM, ExLlamaV3 and SGLang. vLLM with NVFP4/FP8 mixed checkpoint, INT4 PLE offload and MTP reaches 101.6 tok/s at 1k context and 98.9 tok/s at 32k, with prefill around 28k tok/s. ExLlamaV3 EXL3 is faster at short single-user chat (168.9 tok/s at 1k) but fails most requests at 32k concurrency. SGLang with a file-backed n-gram table decodes at 37-51 tok/s. The user notes the 27B incumbent stays in production due to larger KV capacity and reliable tool_choice enforcement.

prompt processing (prefill):

13816.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a prompt-processing regression on Qwen3.6-35B-A3B, dropping from 13816.18 tokens per second to 6858.24 tokens per second between llama.cpp builds 4ebdf2c74 and bed0a8566. Setup is llama.cpp with Q4_K_M GGUF, 32768 context, -ngl 99 across 3x RTX Pro 6000 Blackwell (96 GB each) on a Threadripper PRO 9975WX with 256 GB RAM. The regression appears in llama-completion and llama-server but not in llama-bench, where both builds measure about 15600 t/s at pp18606.

generation:

8.3 tokens/s

prompt processing (prefill):

444.2 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks GLM-4.5-Air with Unsloth UD-Q3_K_XL quant on an RTX PRO 6000 96GB with CPU offload for some experts, using ik_llama.cpp at 16k context with q8_0 KV cache. Setup is ik_llama.cpp with UD-Q3_K_XL and q8_0 KV cache, offloading some expert layers to a Ryzen 9 7950X with 128GB DDR5-3600. At 12288 tokens of context, prompt processing reaches 444.23 t/s and generation 8.25 t/s. The user also compares mainline llama.cpp and an Ubergarm IQ3_KT quant, noting a batch warm-up quirk in the forked llama-sweep-bench.

generation:

104.9 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks DeepSeek V4.1 Flash at 104.86 tokens/s decode on six RTX PRO 6000 Blackwell 96 GB cards. Setup uses B12X linears and experts with PCIe IPC, prefill budget 1536, concurrency 1, no CPU offload. Decode holds 103.37 tokens/s at 32K and 103.84 tokens/s at 128K; TTFT is 2.383 s at 32K and 9.848 s at 128K. The exact 1M-token test (1,048,448 input, 128 forced output) completed in 170.08 s, down from 187.10 s. The final profile shortened 128K TTFT by 15.3%. Server decode rates were 101.5-103.1 tokens/s against client estimates of 103-105 tokens/s.

generation:

112.5 tokens/s

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi-K2.6 at 112.5 tok/s single-stream decode on 8x RTX PRO 6000 Blackwell. Setup is vLLM with fp8 KV cache, TRITON_MLA attention, TP=8, DCP=8, and Eagle3 MTP speculative decoding with 3 tokens using the lightseekorg/kimi-k2.5-eagle3-mla draft model. The 112.5 tok/s is the DCP=1 + MTP=3, 0-context, concurrency-1 cell from an aggregate decode matrix; the page also gives DCP=1 no-MTP (91.5), DCP=4 MTP (101.5), DCP=8 MTP (89.4) and DCP=8 no-MTP (75.5) at the same cell, plus prefill of 7892 tok/s at 8k and 5604 tok/s at ~83k. An erratum notes the old 128k rows were actually ~83k prompt tokens.

generation:

179.5 tokens/s

quant:

EXL3-K3 (EXL3)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash at 179.50 tok/s single-stream decode on two RTX PRO 6000 Blackwell 96GB GPUs. Setup is vLLM with EXL3-K3 quant and FP8 KV cache, TP2 + EP2 + DCP2, DFlash2 speculative decoding, 1,048,576-token context ceiling. Aggregate decode reaches 1,045.47 tok/s at concurrency 16; prefill is 4,107-4,258 tok/s across 8K-128K prompts. Exact one-million-token six-needle retrieval passed 6/6 in 284.859 seconds.

generation:

210.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B NVFP4 at 210 tok/s single-stream on one RTX PRO 6000 Blackwell, using SGLang with the DFlash2 drafter. Setup is SGLang v0.5.20 with a 262,144-token window and 4 slots; the same model without a drafter ran 75 tok/s, and vLLM with DFlash2 ran 160 tok/s. The 27B matrix ran at a 400W power cap. User also reports 607 tok/s aggregate across 4 users, 97s full-window prefill, 73.3% BFCL core, 700/1000 arena score, and 24/40 CAPTCHA puzzles. Flash-Next and an uncensored 27B fine-tune were tested alongside.

generation:

63.7 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 63.7 tok/s single-request decode on 4x RTX PRO 6000 Blackwell GPUs. Setup is an experimental pinned SGLang build with an NVFP4 checkpoint, FP8 E4M3 KV cache, FlashInfer sparse MLA, and 262,144-token context, using tensor parallel TP4/EP4. With EAGLE speculative decoding (adaptive, up to 5 steps) the single-request figure rises to 82.8 tok/s. Eight concurrent requests give 183.8 tok/s aggregate baseline and 182.1 tok/s with EAGLE.

generation:

106.1 tokens/s

prompt processing (prefill):

13036.0 tokens/s

quant:

FP4+FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash at 106.1 tok/s single-request decode on 2x RTX Pro 6000 Blackwell (192GB VRAM). Setup is vLLM with FP4+FP8 quantization and FP8 KV cache, no prompt caching and no speculative decoding, 1,024 output tokens per request. Single-request prefill at 8K context is 13,036 tok/s; decode falls to 96.3 tok/s at 1024K context. System throughput with 8 concurrent requests is 367.8 tok/s at 1K context, 170.0 tok/s at 32K and 56.5 tok/s at 128K.

generation:

246.0 tokens/s

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MiMo-V2.6-Flash-RL at 246 tok/s decode at C1 (90.0 decode steps/s) on two RTX PRO 6000 Blackwell GPUs. Setup is vLLM with FP8 KV cache, TP2 at 0.985 utilization, DFlash drafter with 7 tokens, video off, 1.31M token KV cache. Decode is 42.3 steps/s at C8 and 30.6 at C16; 246 tok/s at C1 on mixed real tasks. Prefill is 9.7K / 9.1K tok/s at 8K / 32K. First start autotunes b12x for about 15 minutes.

generation:

51.0 tokens/s

prompt processing (prefill):

15359.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B IT NVFP4 Turbo at 51 tok/s single-stream decode and 15359 tok/s prefill on an RTX PRO 6000. Setup is vLLM with NVFP4 weights, FP8 KV cache, and 16384 tokens of context, using 18.5 GiB of GPU memory. Batched decode reaches 1244 tok/s and concurrency 6.22 req/s at 1K input / 200 output tokens; the same benchmark on an RTX 5090 performed identically because at 16k context performance is not limited by GPU memory.

generation:

118.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B with MTP speculative decoding at 117.95 tok/s on an RTX Pro 6000 Blackwell, compared to 43.22 tok/s without MTP, a 2.73x speedup. Setup is llama.cpp with Q8_0 GGUF weights, 3000 token context, and 4 speculative tokens. Hardware is an RTX Pro 6000 Blackwell with 96GB VRAM and 92GB system RAM. A separate vLLM run of Gemma 4 31B FP8 with MTP averaged 124.26 tok/s over 10 runs, totaling 4740 tokens in 38.14s.

generation:

77.6 tokens/s

prompt processing (prefill):

14-37 tokens/s

quant:

BF16 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 77-80 tok/s on a single RTX PRO 6000 Blackwell 96GB. Setup is SGLang with BF16 safetensors, FP8 KV cache, EAGLE speculative decoding, and 262144 context length. The decode figure is a range; EAGLE accept rate was 0.94-1.00 and the user claims about 2.2x speedup over non-speculative decoding. Prefill was 14-37 tok/s on 122-token batches.

generation:

60-88 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen 3.8 27B FP8 on a single RTX PRO 6000 Blackwell 96GB with vLLM, shared by a small group of developers for coding agents. Setup uses vLLM with prefix caching and speculative decoding. In its first week the node processed 2B tokens, with two users alone accounting for roughly 950M and 900M tokens. Per-user throughput while sharing the node is 60-88 tok/s. The user reports 0 preemptions, 0 queued requests and 38.6% peak KV cache usage, and notes the node is not yet saturated.

User describes an open-source project (jevm) that replaces the lm_head of Qwen3.8-27B with ordinal cumulative-link heads on several layers, trained with LoRA, to produce five calibrated answers in one prefill for Chinese A-share news screening. No model results are reported: only a smoke test on Qwen3-0.6B has been run, and the 27B is planned for training on 8× RTX PRO 6000. The repo contains label validation and baselines, including a text-blind GBM at 0.55–0.57 AUC on price reactions and 0.68–0.70 on volume/propagation. A plain LoRA plus single-letter-answer baseline is served with vLLM logprobs for comparison.

generation:

60.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~60 t/s on a rented RTX Pro 6000 Blackwell 96GB. Setup uses the Q4_K_M GGUF, which the user notes is not efficient for this card but was chosen because abliterated models that fit the memory are limited. The rental costs about $30/day on Vast, and the user is asking which hardware configuration would offer the best price and versatility for future model generations.

generation:

600.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 600 t/s single request on an RTX Pro 6000 Blackwell. Setup uses the Ninfer engine. User describes it as a drudgework model for read-and-find or code tasks, noting it may use 20x more tokens but is still faster than many local models, and compares it favorably to Cerebras.

prompt processing (prefill):

11352.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on one RTX PRO 6000 Blackwell 96GB, with the official SGLang NVFP4 image cutting time to first token at roughly 254K context from 34.9s to 22.4s and raising prefill from 7,284 to 11,352 tok/s. Setup is the lmsysorg/sglang:dev-qwen38-next-local image with the RadixArk NVFP4 checkpoint, a 262,144-token window and one slot; the previous build streamed the 47.68 GiB per-layer embedding table from NVMe while the official image pins it in host RAM. A separate cache test dropped first-token time from 21.7s cold to 0.44s cached. Long-context recall scored 81/81, BFCL single-turn tool accuracy 85.2% versus 49.5% multi-turn, and tau2-bench telecom completion 68.1% with 41.2% passing all three attempts.