llamaperf

Qwen3.8 125B (6B active) Flash-Next

on AMD Radeon AI PRO R9700 32GB · ExLlamaV3 · 262,144 ctx

AMD hardware
Oct 2, 2026
Throughput
34.7 t/s gen · 863.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
exl3-5.05bpw (EXL3)
KV cache
Q8
System RAM
90 GB

Summary

User reports Qwen3.8-Flash-Next at 863 t/s prefill and 34.7 t/s decode on a single Radeon AI PRO R9700 32GB at 230k context. Setup is ExLlamaV3 (phoenixhaxor ROCm fork, commit cbbef08, patched for gfx12) with exl3 5.05 bpw, Q8 KV cache, 262144 context, chunk size 4096, batch 1. 112 of 512 experts per layer run on the GPU and 400 on CPU with 16 threads; the 102GB bf16 n-gram table is streamed from NVMe. MTP drafting is on with around 53-54% acceptance. The prefill figure covers only the new 101k tokens, the rest came from the prefix cache. User asks whether this is normal for the card or if there is room to tune.