Local model performance on your hardware

generation:

106.6 tokens/s

quant:

BF16 (safetensors)

kv:

BF16

User reports bit-exact speculative decoding on Qwen3.8-27B at 106.6 t/s on short prompts on one RTX PRO 6000 Blackwell Max-Q. Setup is SGLang with BF16 weights and BF16 KV cache, DFlash 8 draft tokens, deterministic inference, Triton attention backend, and three patches including a DeepGEMM fix and PR #42145. Exact decode is 3% behind vLLM on short prompts, ahead at 2.2k tokens (73.9 vs 67.4 t/s), and 24% behind at 128k (42.6 vs 56.1 t/s). Prefill is slower: 68.5 s to first token on a 128k prompt versus 44.3 s on vLLM. Deterministic mode without the PR was 18.0 t/s, against 26.4 stock.

generation:

130-180 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8 27B at 130-180 t/s on an RTX Pro 6000 Blackwell Max-Q. Setup is NInfer with NVFP4 weights, DFlash2 and MTP-7 speculative decoding, on Windows 10. The user says parallel decoding shows hardly any slowdown and concurrent capabilities are on par with vLLM, and notes the backend is new and still missing features such as advanced sampling parameters and broader model support.

generation:

249.1 tokens/s

prompt processing (prefill):

17982.0 tokens/s

kv:

fp8

User reports DeepSeek-V4.1-Flash at 249.1 tok/s single-stream decode and 17,982 tok/s prefill on four RTX PRO 6000 Max-Q cards. Setup is vLLM with bfloat16 weights, fp8 KV cache, TP4/DCP1, RAM Engram tables, adaptive DSpark K7 speculation, 4096-token budget, 32 slots and a 131,072 context cap. C8 aggregate output is 805.5 tok/s and the C1 verifier rate is 98.58 steps/s; the previous Karmic Kraken release measured 230.6 tok/s, 748.3 tok/s, 17,503 tok/s and 89.87 steps/s respectively.

generation:

35.7 tokens/s

prompt processing (prefill):

38.3 tokens/s

quant:

FP4-FP8 (GGUF)

User reports DeepSeek-V4-Flash on 2x RTX PRO 6000 Blackwell Max-Q 96GB (192GB total), with prompt processing 36.5-39.4 t/s and token generation 34.1-41.7 t/s. Setup is llama.cpp wip/deepseek-v4-support branch with a native FP4 + FP8 GGUF (146GB, 4.39 bits per weight), ctx-size 8192, batch 4096, Flash Attention disabled, no speculative decoding, single request, short prompts of 14-50 tokens. GPU utilization was 30-40% burst; power draw was 97.8W on GPU0 and 115W on GPU1.

generation:

43.6 tokens/s

prompt processing (prefill):

262.0 tokens/s

quant:

IQ2_XXS + Q2_K (GGUF)

User reports DeepSeek V4 Flash at 43.6 tok/s generation on a short prompt and 262 tok/s prefill on a 20K-token prompt on an RTX PRO 6000 Blackwell Max-Q. Setup is DwarfStar 4 (ds4.c) with an IQ2_XXS + Q2_K quant, single session, no MTP or speculative decoding. Generation drops to 31.4 tok/s at 50K context; peak VRAM observed 93,142 MiB of 95,593 MiB (95%).

prompt processing (prefill):

16384.0 tokens/s

User benchmarks llama.cpp PR #22675 for Mamba-2 prefill acceleration on Nemotron-Nano-9B-v2, with pp16384 tok/s reported. The PR is tested at various ub values. The user also mentions DeepSeek-V4-Flash-UD-IQ2_XXS on an M4 Max for a Metal FWHT kernel PR, but the primary benchmarked model is Nemotron-Nano-9B-v2.

generation:

44.0 tokens/s

prompt processing (prefill):

564.0 tokens/s

quant:

UD-Q8_K_XL

User benchmarks multiple quants on a Bosgame M5 with an RTX PRO 6000 Max-Q eGPU. Setup is a mixed GPU configuration with CUDA0 and ROCm0, using a DSpark drafter ported from a closed PR. UD-Q8_K_XL (150.8 GiB) reaches 44.0 t/s decode and 564 t/s prefill with draft acceptance 0.535. UD-Q4_K_XL (144.4 GiB) reaches 48.4 t/s decode and 585 t/s prefill with draft acceptance 0.532. UD-Q2_K_XL (90.2 GiB) reaches 59.5 t/s decode and 1513 t/s prefill with no drafter.

generation:

96.8 tokens/s

quant:

groupwise-int

kv:

Q8

User reports a long coding-agent session with DeepSeek Harness at a weighted 96.8 t/s streaming rate and a median of 104.4 t/s. The model is a groupwise-int artifact, not NVFP4. Context compaction worked well.

generation:

58.0 tokens/s

quant:

BF16

User reports decode speed rising from 18 tok/s to 58 tok/s with DFlash2 + XQA versus Flash2 + FA2 at 192K context. Lossless BF16 weights and KV.

generation:

177.0 tokens/s

quant:

NVFP4

User compares Flash-Next against a dense 27B model. Flash-Next is faster and mechanically flawless, with zero failures on strict JSON, injection resistance and SLAs, and wins the high-reasoning spatial and code-gen tier with better failure modes. The dense 27B still wins sustained multi-step symbolic work such as bug-fixing, math proofs and abstract puzzles. On that symbolic work Flash-Next shows a new failure shape: it promises the deliverable, declares "done", and outputs nothing. Both models use the same reasoning_effort knob with radically different semantics, so Flash-Next is not a drop-in replacement but a conditional promotion.

generation:

240.0 tokens/s

quant:

NVFP4

User reports Qwen3.8-Next-Flash at ~240 t/s decode on a single RTX Pro 6000 Max-Q at 300W. Setup is a patched sglang with the RadixArk/Qwen3.8-Flash-Next-NVFP4 checkpoint, using FP8 quantization of the lm head and layers, kernel tuning, and MTP config tuning. The user notes a theoretical bandwidth limit of ~280 t/s.

User benchmarks DeepSeek V4 Pro GGUF at 794 GB on an Epyc 9374F with 12x96 GB DDR5 and an RTX PRO 6000 Max-Q. Setup is llama.cpp on a branch with expert offloading. Prompt processing ranges from 192 t/s at 8K context to 66 t/s at 1M context, and generation ranges from 11.73 t/s to 5.83 t/s. RAM usage is 69.3% of 1152 GB and VRAM usage is 78986 MiB of 96 GB, with power around 500 W during prompt processing. The user notes mainline llama.cpp issues with memory waste, a broken quantized KV cache, and bugs with prompt cache reuse.

generation:

7.1 tokens/s

prompt processing (prefill):

46.2 tokens/s

User benchmarks at various context depths on an Epyc 9374F. GPU VRAM usage is 90815MiB/97887MiB and RAM usage is 907.5GB/1152GB. Power draw is about 100W prefill and 150W token generation for the GPU, and about 400W for the CPU and motherboard. The original model files are used with no conversion.