Use cases
vision
Summary
User reports ~3700 t/s prefill with Qwen3.8 Flash-Next fp8-iq4r on a single RX 9070 XT, with experts placed in RAM and a ~57GB host pool.
KV cache uses fp8_v4 and --max-num-batched-tokens is set to 4096, which gave the best prefill results. The card is power limited during prefill at over 300W, and PCIe5 x16 is nearly saturated; gen 4 drops prefill to ~2000 t/s.
Prefill stays flat across context while decode decays 18-19% over 100k tokens, with a worst case of ~45 t/s at 240k context. Vision is fully working. The user has 96GB of DDR5 and is still working on decode speed.