Qwen3.8 27B
on NVIDIA RTX 5090 · NInfer · 262,144 ctx
Sep 7, 2026
Summary
User reports an aggregate 880 t/s at 6 parallel requests, peaking at 967 t/s, and over 200 t/s single stream with MTP speculative decoding.
Prefill runs at about 5,950 t/s. Weights take 16.8 GiB, leaving about 13 GiB for KV cache.
Benchmarks are HumanEval+ 152/164 and AIME25+AIME26 55/60, identical to int4 but 1.56x-1.98x faster. The run requires Blackwell FP4 cores and a 6-line patch not yet upstream.