llamaperf
Oct 8, 2026
Throughput
64.0 t/s gen · 1200.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
AWQ W4A16 (AWQ)
VRAM reported
32 GB

Summary

User reports Qwen3.8-Flash-Next at ~64 tokens/s single-stream decode and ~1,200 tokens/s prefill on four AMD Radeon PRO V620 cards (32 GB each, gfx1030) in one box. Setup is a vLLM fork with AWQ W4A16 backbone plus a quantised int4 n-gram table, tensor parallel across all four cards, MTP off and cards uncapped; the n-gram table is CPU-offloaded. At the production 140 W setting decode is ~59.5 tokens/s, and with MTP=3 it ranges ~60-72 tokens/s. Model files total ~105 GB of downloads.