NVIDIA RTX Pro 4500 Blackwell 32GB

generation:

60-78 tokens/s

quant:

FP4-Local QAD (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next FP4-Local at roughly 60-78 tok/s around 40-55k real context on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5. Setup is Strata 0.1.45 with NVIDIA FP4-Local QAD weights, INT8 KV cache at 200k context, a ~38GB RAM expert tier with adaptive VRAM expert cache, SSD-backed PLE and MTP speculative decoding (spec=6). Cold 43k prompt prefill was about 1,890 tok/s with up to ~90% GPU expert-cache hit. BFCL-v4 quick test scored 85% overall and 92% on the 50 single-turn tool-selection tasks.

prompt processing (prefill):

1680.0 tokens/s

quant:

NVFP4 (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next NVFP4 running on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5 system RAM, reaching up to ~80 tok/s at ~50k context and ~60-67 tok/s at ~188k warm context. Setup is the Strata NVFP4 fork with NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding, and W4A8 prefill on Blackwell, served through Strata's OpenAI-compatible server. A cold 189k full prompt measured ~1,680 tok/s prefill and ~53 tok/s decode. User calls the result impressive for a single 32GB GPU with only 64GB system RAM.

generation:

36.5 tokens/s

prompt processing (prefill):

909.0 tokens/s

quant:

PTQ1_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ternary-Bonsai-2-27B at PTQ1_0 running on a 32 GB RTX PRO 4500 Blackwell via llama.cpp, with 36.5 tok/s decode and 909 tok/s prefill at 32,768 tokens of context in a single stream. Setup uses the 6.58 GB ternary GGUF with a 131,072-token context window; the Q8_0 mmproj vision tower still works in the ternary GGUF. At 131,072 tokens of context the single stream drops to 14.1 tok/s decode, 630 tok/s prefill and 207 s time to first token. Two and four concurrent streams at 32,768 tokens give 12.4 and 4.6 tok/s decode per stream, and the concurrent cells at 131,072 tokens returned HTTP 500. The user notes decode collapses with concurrency, unlike EXL3 rows which hold a flat decode rate.

generation:

27.9 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B MoE at 27.9 t/s token-weighted decode on an RTX PRO 4500 Blackwell 32GB with 64GB system RAM. Setup is a llama.cpp fork with UD-Q3_K_XL GGUF, 240,000 context, F16 KV cache, 16 of 48 routed-expert layers on GPU and 32 on CPU, disk-backed PLE, and MTP disabled. Real prompt chunks prefill around 150 t/s with a peak of 201. Decode stayed flat from 116K to 148K context (27.7, 28.5, 27.5 t/s across the three bands). The user notes the sparse-QSA CUDA patch is experimental and unreviewed, and that these are warm-cache workload numbers rather than a controlled benchmark.

generation:

45.2 tokens/s

prompt processing (prefill):

2022.5 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B IQ4_XS on an RTX Pro 4500 Blackwell 32GB with llama.cpp b9007, reaching 2022.54 t/s prompt processing and 45.19 t/s generation. The same card also runs Qwen3.6 35B-A3B MXFP4 at 5507.10 t/s prompt processing and 159.81 t/s generation, along with Gemma4 26B-A4B MXFP4, Ernie 4.5 21B-A3B MXFP4, Nemotron Cascade 2 30B-A3B MXFP4, Tesselate OmniCoder 9B Q8, Qwen3.5 4B Q4_K, Qwen3.5 9B UD Q4_K_XL and GLM 4.7 Flash MXFP4. Compared with an RTX 5090, the 5090 is 60-70% faster at 2-3x power. User is happy with the card for 24/7 use.