llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 2× NVIDIA RTX 5090 · ExLlamaV3 · 262,144 ctx

NVIDIA hardware
Sep 25, 2026
Throughput
291.1 t/s gen
Quant
2.50bpw EXL3 (EXL3)
KV cache
8-bit
VRAM reported
32 GB

Use cases

codinglong-contexttool-use

Summary

User reports Qwen3.8-Flash-Next at 291.1 t/s single-stream decode on code and 273.5 t/s on prose, on two RTX 5090 32 GB GPUs with layer split [30, 30]. Setup is ExLlamaV3 v1.5.0 under TabbyAPI with a 2.50 bpw EXL3 pack, 8-bit KV cache, 262,144-token context and a 983,040-token page pool, using MTP draft depth 3 up to 4 jobs and 2 at 5 to 8 jobs. Per-stream decode falls to 153.2 t/s on code and 156.8 t/s on prose at 4 streams, and 103.4 t/s on code and 103.6 t/s on prose at 8 streams. Aggregate decode across concurrent streams is 292 t/s code and 272 t/s prose at 1 stream, 615 t/s code and 627 t/s prose at 4 streams, and 822 t/s code and 831 t/s prose at 8 streams. Time to first token ranges from 0.13 to 0.14 s at 1 stream to 0.76 s at 8 streams. Cold prefill holds its rate to the top of the window: 199,844 tokens in 18.8 s and 240,047 tokens in 22.8 s. Quality runs on the same configuration give GSM8K 5-shot (n=500) 0.978, SWE-bench Verified 397/500 resolved (79.4%), and tool-eval-bench 84.0 +/- 2.4.