llamaperf

NVIDIA RTX Pro 4000 Blackwell

NVIDIA · 24GB · 7 reports

Is this your card?

Engines people use on it: NInfer 2 (reports) · llama.cpp 2 (reports) · Ollama 1 (reports) · SGLang 1 (reports)

Run models on your NVIDIA RTX Pro 4000 Blackwell? Add your numbers.

Your coding agent can run a speed test on your machine and send it here with a free account. Or paste a result you've already posted.

Reported performance on the NVIDIA RTX Pro 4000 Blackwell

These reports mix GPU counts and memory setups, so a model listed here may have needed extra cards or CPU offloading. The calculator estimates which models fit in 24 GB of VRAM. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Qwen3.8 27B

2× NVIDIA RTX Pro 4000 Blackwell · NInfer · 39,843 ctx

NVIDIA hardware
generation:
111.1 tokens/s
prompt processing (prefill):
5634.0 tokens/s
quant:
QUASAR mixed NVFP4/FP8
codingmath

User benchmarks Qwen3.8-27B at 155.9 t/s decode on a ~40k-token prompt using two RTX PRO 4000 Blackwell 24GB cards in tensor parallel 2. Setup is NInfer v0.9.0 (ninfer-serve, CUDA) with a QUASAR mixed NVFP4/FP8 artifact, int8 KV cache, 262144-token context and DFlash2 speculative decoding with 7 draft tokens; GPUs are power-capped to 145 W each. A second long-context run gave 163.8 t/s decode and 5590 t/s prefill; prose decode was 111.1 t/s and short reasoning tasks averaged 204.7 t/s with 10/10 correct. A ~9k-token prompt had a median TTFT of 1.31 s. Timings are client-side, one request at a time.

Nov 9, 2026
Tone: positiveNVIDIA hardware
generation:
30-35 tokens/s
quant:
2.25 bpw

User reports a DeepSeek model at 2.25 bpw running at around 30-35 tokens/s on two RTX PRO 4000 Blackwell cards. The machine is a Xeon W-2245 workstation with 256GB of DDR4-2666 ECC RAM in 8 x 32GB sticks. No engine, context length or KV cache setting is stated. The speed is given only as a range, so no single figure is recorded.

Nov 4, 2026

Qwen3.8 27B

2× NVIDIA RTX Pro 4000 Blackwell · llama.cpp · 256,000 ctx

Tone: positiveNVIDIA hardware
generation:
68.0 tokens/s
quant:
Q8 (GGUF)
agentictool-usecodinglong-contextvisionsummarization

User reports a fully local voice and chat assistant running Qwen3.8-27B (dense, 27.3B, Q8) at roughly 68 tok/s decode and ~1.5K tok/s prefill on 2x RTX PRO 4000 Blackwell 24GB, with an RTX 5060 Ti 16GB handling display and helper models. Setup is a patched llama.cpp build for Blackwell with speculative decoding (MTP, ~62% draft acceptance) and a vision projector, serving 256K context across 2 concurrent slots. The machine is an AMD Threadripper 9960X with 128 GB DDR5 ECC on Ubuntu 26.04, drawing ~820 W across GPUs and CPU. The user built a custom Python proxy (~22 modules) in front of llama-server with tool-calling, context compaction, a document workspace, and six MCP servers (filesystem, SearXNG web search, ChromaDB RAG with bge-m3, reminders, documents, sandboxed code editor). A voice pipeline with wake word, faster-whisper STT and Piper TTS is currently paused. The user asks for recommendations and comments.

Oct 16, 2026
NVIDIA hardware
generation:
13.6 tokens/s
prompt processing (prefill):
338.4 tokens/s
quant:
MXFP4 (GGUF)

User reports DeepSeek-V4-Flash at 13.61 t/s generation and 338.41 t/s prompt eval on 2 x RTX PRO 4000 with 48 GiB VRAM and 160 GiB system RAM. Setup is llama.cpp with an MXFP4/Q8_0 GGUF (about 146 GiB) at 32768 context, using --fit on and --no-mmap. A llama-batched-bench sweep from 8K to 1M context shows prompt throughput falling from 448.57 t/s to 65.97 t/s and generation from 19.20 t/s to 6.56 t/s; without hc-ops the same sweep was slower (322.50 t/s prompt, 14.62 t/s generation at 8K). KLD against the base logits is 0.422277 mean for the Q2 quant and 0.115089 for the Q4 quant.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
71.4 tokens/s
quant:
AWQ (AWQ)
kv:
fp8_e5m2
codinglong-context

User benchmarks Qwen2.5-32B-Instruct-AWQ at 71.37 tok/s median decode on 2x RTX PRO 4000 Blackwell. Setup is SGLang 0.5.19 with AWQ weights, NGRAM speculative decoding (K=5), tensor parallelism 2 over PCIe Gen4, and FP8 KV cache. NGRAM speculative decoding gives a 1.28x median speedup over the 55.60 tok/s baseline, with high inter-prompt variance (std dev 19.18 tok/s). Per-domain decode ranges are 65-107 tok/s for JSON, 60-226 tok/s for code, and 58-82 tok/s for prose. FP8 KV cache alone measured 54.77 tok/s, a 1.5% regression.

Sep 26, 2026
Tone: positiveNVIDIA hardware
generation:
67.0 tokens/s
prompt processing (prefill):
785.0 tokens/s
kv:
Q8

User reports 67 t/s with MTP3 speculative decoding enabled, against 24.4 tok/s without MTP. Setup uses an INT8 KV cache with group-64. The figure comes from a 128K NIAH benchmark with 130,048 prompt tokens, where MTP acceptance was 100% on a deterministic answer. Roughly 727 MiB of VRAM was left.

Sep 7, 2026
Tone: positiveNVIDIA hardware
generation:
33.9 tokens/s

User reports Qwen3.8:27b at 33.91 t/s on 2x RTX PRO 4000s and an RTX PRO 2000, up from 12.2 t/s. Splitting the GPUs across VMs produced the gain. User also reports muse-glimmer:30B at 22.61 t/s on the same setup, up from 14.3 t/s.

Sep 7, 2026

Get a weekly email of new NVIDIA RTX Pro 4000 Blackwell reports and newly released models that fit it.

Email me new reports

Reported configurations

Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.

Model / hardwareQuant / engineReported contextGeneration
Qwen3.8 27B
2× NVIDIA RTX Pro 4000 Blackwell
QUASAR mixed NVFP4/FP8
NInfer
39,843111.1 tokens/s
Qwen3.8 27B
2× NVIDIA RTX Pro 4000 Blackwell
Q8
llama.cpp
256,00068.0 tokens/s
DeepSeek V4 Flash 284B (13B active)
2× NVIDIA RTX Pro 4000 Blackwell
MXFP4
llama.cpp
32,76813.6 tokens/s
Qwen2.5 32B
2× NVIDIA RTX Pro 4000 Blackwell
AWQ
SGLang
Not reported71.4 tokens/s
Qwen3.8 27B
NVIDIA RTX Pro 4000 Blackwell
Not reported
NInfer
128,00067.0 tokens/s
Qwen3.8 27B
NVIDIA RTX Pro 4000 Blackwell
Not reported
Ollama
Not reported33.9 tokens/s