Qwen3.8 125B (6B active) Flash-Next
AMD Radeon AI PRO R9700 32GB · ExLlamaV3 · 262,144 ctx
- generation:
- 34.7 tokens/s
- prompt processing (prefill):
- 863.0 tokens/s
- quant:
- exl3-5.05bpw (EXL3)
- kv:
- Q8
User reports Qwen3.8-Flash-Next at 863 t/s prefill and 34.7 t/s decode on a single Radeon AI PRO R9700 32GB at 230k context. Setup is ExLlamaV3 (phoenixhaxor ROCm fork, commit cbbef08, patched for gfx12) with exl3 5.05 bpw, Q8 KV cache, 262144 context, chunk size 4096, batch 1. 112 of 512 experts per layer run on the GPU and 400 on CPU with 16 threads; the 102GB bf16 n-gram table is streamed from NVMe. MTP drafting is on with around 53-54% acceptance. The prefill figure covers only the new 101k tokens, the rest came from the prefix cache. User asks whether this is normal for the card or if there is room to tune.