llamaperf
Oct 24, 2026
Throughput
236.0 t/s gen · 8300-8900 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
fp8-iq4r-moe
KV cache
fp8
System RAM
122 GB
VRAM reported
32 GB

Summary

User reports Qwen3.8-Flash-Next at 236 tok/s single-stream aggregate on four R9700 32 GB cards at TP4, with 1,192 tok/s at 16 concurrent streams. Setup is radiance 1.11.0 with the fp8-iq4r-moe container (113.6 GiB, 4-bit experts, int8 trunk, bf16 protected experts), fp8 KV cache, MTP head depth 3, 262144 max model length, all weights in VRAM and the 47.7 GiB n-gram table pinned in host RAM. Decode step is 11.27 ms against a 11.2 ms reference; prefill measured 8.4K t/s at 4.4K and 70K prompt tokens, and single stream with thinking on reached 197 tok/s over 4,096 generated tokens. Concurrency rows are thinking off, 512 tokens each, wall-clock over the whole batch; a 2x TP2 attempt lost.