Local model performance on your hardware

generation:

64.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

quant:

AWQ W4A16 (AWQ)

User reports Qwen3.8-Flash-Next at ~64 tokens/s single-stream decode and ~1,200 tokens/s prefill on four AMD Radeon PRO V620 cards (32 GB each, gfx1030) in one box. Setup is a vLLM fork with AWQ W4A16 backbone plus a quantised int4 n-gram table, tensor parallel across all four cards, MTP off and cards uncapped; the n-gram table is CPU-offloaded. At the production 140 W setting decode is ~59.5 tokens/s, and with MTP=3 it ranges ~60-72 tokens/s. Model files total ~105 GB of downloads.

generation:

24.6 tokens/s

prompt processing (prefill):

492.0 tokens/s

quant:

Q4_K_XL (GGUF)

User reports Swift-Qwen3.8-27B at 24.6 t/s decode and 492 t/s prefill on a single Radeon PRO V620 32GB. Setup is a llama.cpp fork for gfx1030 with Q4_K_XL GGUF (16.6 GiB), flash attention, 128k context, card at 300 W. Decode falls to 16.9 t/s at 128k depth; with an MTP drafter it reaches 48.0 t/s at depth 0 and 44.1 t/s at 120k, while DFlash2 peaks at 76 t/s on short mixed work.

generation:

42.0 tokens/s

quant:

Q4

User reports Qwen 3.8 27B at up to 42 t/s on 2x Radeon Pro V620 32GB cards at 41k context. Setup is vLLM 0.27.1 with a Q4 quant and tensor parallelism of 2. The first working run gave 11-18 t/s; after optimization the two-card combo reached 42 t/s. A four-card TP=4 config for a 122B MoE reaches 56-59 t/s. A successor repo runs Qwen3.8-Flash-Next at ~100 t/s single-stream on 4x V620, versus 29-30 t/s for llama.cpp on the same box.

generation:

62-68 tokens/s

quant:

AWQ-W4A16 (AWQ)

User reports Qwen3.8-Flash-Next at 62-68 tok/s decode on 3x Radeon PRO V620 32GB in pipeline parallel PP=3. Setup is vLLM with an AWQ-W4A16 checkpoint, int4 n-gram sidecar in host RAM, MTP k=2 speculative decoding, 262K context, 160 W power cap per card. Prefill measured at 1152 / 1705 / 1936 / 1933 tok/s for 4K / 16K / 64K / 130K prompts; a 200 W cap raises prefill to 1306 / 1943 / 2221 / 2254 tok/s with decode unchanged. Multi-stream decode without MTP reaches 157 tok/s at 4 streams and 242 at 8; with MTP 159 at 4 and 91 at 8.

generation:

60-100 tokens/s

quant:

W4A16 (safetensors)

User reports Qwen3.8-Flash-Next at 60 to 100 t/s decode and over 3000 t/s prefill on a rig with 8x Radeon Pro V620 (256 GB VRAM). Setup is a custom vLLM fork with W4A16 routed experts and BF16 elsewhere, pipeline parallel 4, MTP enabled with 3 token drafting, concurrency 1. User says llama.cpp managed only around 350-450 t/s prefill with the same model, and that the new setup is about 800% faster.

generation:

70.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

W4A16 (AutoRound)

User reports Qwen3.8-Flash-Next at roughly 2,000 t/s prefill and 70 t/s decode on 4x Radeon Pro V620 32GB. Setup is Intel Autoround W4A16 weights running locally with the OMP harness, which the user believes made a big difference. The model ran for around 3 hours, mostly driving two browsers at once to test and fix a generated HTML/JS 3D game.

generation:

70.0 tokens/s

prompt processing (prefill):

1300.0 tokens/s

quant:

W4A16 (AutoRound)

mtp (multi-token prediction):

on

User reports Qwen3.8 Flash-Next at 70 t/s generation and 1.3k t/s prefill on 4x V620 (128GB total VRAM) at 128k+ context. Setup is a vLLM fork with AutoRound W4A16 quantization and MTP-2 speculative decoding, on an EPYC 7452 with 256GB DDR4. Prose generation runs at 60 t/s. Power draw is 700-900W during prefill and 500-600W during decode. User was initially disappointed with Qwen3.8 27B speeds but is satisfied with Flash-Next.

generation:

70.0 tokens/s

prompt processing (prefill):

1386.0 tokens/s

quant:

W4A16 (AutoRound)

User reports Qwen3.8 Flash-Next at 70.0 t/s generation and 1,386 t/s prompt processing on 4 Radeon Pro V620 GPUs at 32K context on the coding suite. Setup is vLLM with the Intel/Qwen3.8-Flash-Next-W4A16-AutoRound quant, using a custom RDNA vLLM fork. At 64K context the coding suite reached 72.5 t/s generation and 1,369 t/s prompt processing; at 128K it reached 68.3 t/s and 1,296 t/s. The regular suite measured 57.9, 59.5, and 56.1 t/s generation at 32K, 64K, and 128K respectively.

generation:

35.4 tokens/s

prompt processing (prefill):

355.4 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

f16

User benchmarks Muse Glimmer 30B on AMD v620 GPUs, with Q6 on one GPU at 355.38 t/s prefill and 35.38 t/s generation. Setup is a tensor split across two GPUs: Q6 gives 472.27 t/s prefill and 36.32 t/s generation, Q8 gives 550.04 t/s prefill and 26.55 t/s generation. Speculative decoding uses a dflash draft model.

generation:

21.1 tokens/s

prompt processing (prefill):

276.2 tokens/s

quant:

IQ3_XXS

kv:

FP16

User reports DeepSeek-V4-Flash-0731 at 21.06 t/s continuous generation and 30.59 t/s sustained accepted generation on 4x AMD Radeon Pro V620 with 128 GB total. Setup is the unsloth IQ3_XXS quant with speculative decoding using a DSpark Q8 drafter on CPU RAM and 2 draft tokens. Prompt ingestion is 276.23 t/s at 32K and 379.01 t/s at 4K. The user calls the model at q3_xxs "fucking stupid" and lobotomized, and notes there is not enough VRAM for the drafter, which affects TG speeds.

generation:

30.0 tokens/s

prompt processing (prefill):

1000-1300 tokens/s

quant:

Q8

User reports Qwen3.8 27B at Q8 running at 30 t/s with 1-1.3k t/s prefill on a Threadripper 3975WX with four Radeon Pro V620 cards. User measures electricity rather than speed: about 0.8-0.85 kW during inference and 0.15-0.2 kW idle, which at 0.32 dollars per kWh works out to roughly 55-60 dollars a month for six hours of inference a day. User calls 30 t/s pretty slow and reports 3.5k prefill and 70-80 t/s on Qwen3.6 35B instead.

generation:

14.3 tokens/s

prompt processing (prefill):

364.0 tokens/s

quant:

Q5_K_P (GGUF)

kv:

Q4

User benchmarks Qwen3.8-27B and Gemma4-26B-A4B on an AMD V620 using llama.cpp with ROCm and Vulkan. Qwen results are shown; Gemma results appear in a table but are not extracted as the primary model.