Local model performance on your hardware

generation:

64.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

quant:

AWQ W4A16 (AWQ)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~64 tokens/s single-stream decode and ~1,200 tokens/s prefill on four AMD Radeon PRO V620 cards (32 GB each, gfx1030) in one box. Setup is a vLLM fork with AWQ W4A16 backbone plus a quantised int4 n-gram table, tensor parallel across all four cards, MTP off and cards uncapped; the n-gram table is CPU-offloaded. At the production 140 W setting decode is ~59.5 tokens/s, and with MTP=3 it ranges ~60-72 tokens/s. Model files total ~105 GB of downloads.

generation:

42.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at up to 42 t/s on 2x Radeon Pro V620 32GB cards at 41k context. Setup is vLLM 0.27.1 with a Q4 quant and tensor parallelism of 2. The first working run gave 11-18 t/s; after optimization the two-card combo reached 42 t/s. A four-card TP=4 config for a 122B MoE reaches 56-59 t/s. A successor repo runs Qwen3.8-Flash-Next at ~100 t/s single-stream on 4x V620, versus 29-30 t/s for llama.cpp on the same box.

generation:

70.0 tokens/s

prompt processing (prefill):

1300.0 tokens/s

quant:

W4A16 (AutoRound)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 70 t/s generation and 1.3k t/s prefill on 4x V620 (128GB total VRAM) at 128k+ context. Setup is a vLLM fork with AutoRound W4A16 quantization and MTP-2 speculative decoding, on an EPYC 7452 with 256GB DDR4. Prose generation runs at 60 t/s. Power draw is 700-900W during prefill and 500-600W during decode. User was initially disappointed with Qwen3.8 27B speeds but is satisfied with Flash-Next.

generation:

70.0 tokens/s

prompt processing (prefill):

1386.0 tokens/s

quant:

W4A16 (AutoRound)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 70.0 t/s generation and 1,386 t/s prompt processing on 4 Radeon Pro V620 GPUs at 32K context on the coding suite. Setup is vLLM with the Intel/Qwen3.8-Flash-Next-W4A16-AutoRound quant, using a custom RDNA vLLM fork. At 64K context the coding suite reached 72.5 t/s generation and 1,369 t/s prompt processing; at 128K it reached 68.3 t/s and 1,296 t/s. The regular suite measured 57.9, 59.5, and 56.1 t/s generation at 32K, 64K, and 128K respectively.