Local model performance on your hardware

generation:

43.0 tokens/s

prompt processing (prefill):

1539.0 tokens/s

quant:

AWQ W4A16 (AWQ)

User reports Qwen3.8-Flash-Next at 43 tok/s decode and 1,539 tok/s prefill on a Strix Halo 128GB at a 5.6k prompt. Setup is a vLLM fork with the libstrix plugin, AWQ W4A16 weights, 16k prefill chunks and MTP 2, single stream. The 102 GB n-gram table is streamed from disk and prefetched into the page cache. At 26.9k and 100.5k prompts decode is 43 and 40 tok/s with prefill 1,617 and 1,572 tok/s. Before this work the same setup did 684-806 tok/s prefill and about 18 tok/s decode. KLD against stock is 0.008-0.019 with 97-98% top-1 agreement, perplexity within 0.6%, and GSM8K 93 vs 92.

generation:

32.6 tokens/s

prompt processing (prefill):

3581.0 tokens/s

quant:

bf16 (BF16)

kv:

bf16

User reports Qwen3.6-35B-A3B at 3,581 t/s prefill and 32.6 t/s decode at 2k context on a Strix Halo 128GB APU, comparing stock vLLM against the same vLLM with the libstrix HIP kernel plugin. Setup is vLLM with bf16 weights and bf16 KV cache, single stream on one APU; the plugin installs via pip and only activates on ROCm gfx1151. At 64k the same run goes from 260 to 2,617 t/s prefill and 12.8 to 26.9 t/s decode; at 128k with int8 KV it reaches 1,751 t/s prefill and 26 t/s decode, and a 256k prompt prefills in 241 s versus 7,568 s stock. KLD against stock is 0.001-0.006 with 97-99% top-1 agreement, and the user notes decode gains are absent where the replaced kernels are not the bottleneck.

generation:

86.6 tokens/s

quant:

BF16 (GGUF)

User benchmarks SGLang 0.5.17, vLLM 0.26.0 and llama.cpp b10333 on a 128GB GMKtec EVO-X3 with Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151). On Qwen3.5-0.8B BF16, custom vLLM reached 86.59 tok/s at one client and 311.59 tok/s aggregate at eight clients, beating SGLang and llama.cpp at every concurrency. Setup used ROCm 7.14, 4096-token API limit, deterministic 598-638 token prompts with 64 output tokens, prefix caching disabled. vLLM won all 20 matched concurrency points against SGLang across Qwen3.5-0.8B, Qwen3-8B, Qwen3-Coder-30B-A3B and a shared-prefix workload. For Qwen3.5-122B-A10B, custom llama.cpp with UD-Q4_K_XL reached 13.57 tok/s at one client and official llama.cpp 17.88 tok/s aggregate at two clients. DeepSeek V4 Flash on a specialised Lucebox ROCmFP3 service reached 23.70 tok/s median.