Local model performance on your hardware

generation:

130-180 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8 27B at 130-180 t/s on an RTX Pro 6000 Blackwell Max-Q. Setup is NInfer with NVFP4 weights, DFlash2 and MTP-7 speculative decoding, on Windows 10. The user says parallel decoding shows hardly any slowdown and concurrent capabilities are on par with vLLM, and notes the backend is new and still missing features such as advanced sampling parameters and broader model support.

generation:

96.8 tokens/s

quant:

groupwise-int

kv:

Q8

User reports a long coding-agent session with DeepSeek Harness at a weighted 96.8 t/s streaming rate and a median of 104.4 t/s. The model is a groupwise-int artifact, not NVFP4. Context compaction worked well.