Qwen3.8 27B
NVIDIA RTX Pro 5000 Blackwell 72GB · vLLM · 262,144 ctx
- generation:
- 140.7 tokens/s
- quant:
- NVFP4
- kv:
- fp8
User benchmarks Qwen3.8 27B at 140.7 tok/s single-stream decode in NVFP4 on an RTX PRO 5000 72GB, up from 83.6 tok/s in FP8. Setup is vLLM 0.29.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, chunked prefill and prefix caching, at 262144 context. Aggregate throughput is 300.0 tok/s at 8 concurrent streams and 321.0 at 16. A 200K prompt gives 85.7 s TTFT and 94.3 tok/s decode in NVFP4 versus 130.8 s and 33.2 tok/s in FP8. Quality on 300 items was 255/300 NVFP4 against 254/300 FP8. In the wider 800-test agentic coding leaderboard the local FP8 run scored 675/800 at 31.8 tok/s.