Local model performance on your hardware

generation:

43.0 tokens/s

prompt processing (prefill):

1539.0 tokens/s

quant:

AWQ W4A16 (AWQ)

User reports Qwen3.8-Flash-Next at 43 tok/s decode and 1,539 tok/s prefill on a Strix Halo 128GB at a 5.6k prompt. Setup is a vLLM fork with the libstrix plugin, AWQ W4A16 weights, 16k prefill chunks and MTP 2, single stream. The 102 GB n-gram table is streamed from disk and prefetched into the page cache. At 26.9k and 100.5k prompts decode is 43 and 40 tok/s with prefill 1,617 and 1,572 tok/s. Before this work the same setup did 684-806 tok/s prefill and about 18 tok/s decode. KLD against stock is 0.008-0.019 with 97-98% top-1 agreement, perplexity within 0.6%, and GSM8K 93 vs 92.

generation:

35.0 tokens/s

prompt processing (prefill):

1888.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 35.0 t/s short-context decode and 1888 t/s prefill on a Strix Halo 128GB (Ryzen AI Max+ 395) box. Setup is Gufo with UD-Q4_K_XL GGUF weights (103.69 GiB across four shards), GPU only, no MTP, FCLK locked at 2000 MHz, ROCm 7.2.3. At 16K context decode is 34.1 t/s. Stock llama.cpp ROCm HIP on the same machine gave around 568 t/s prefill and 23-24 t/s decode. The model is the ~125B MoE with ~6B active plus n-gram embedding and MTP modules, fitting in the 128 GB pool with about 117 GiB GTT reported.

generation:

24.5 tokens/s

prompt processing (prefill):

1228.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 24.5 t/s decode and 1,228 t/s prefill on a single Strix Halo with their Gufo fork. Setup is Gufo with the IQ1_M GGUF resident, HTTP with real chat prompts, greedy decoding and thinking off; DSpark speculation reaches 30.7 t/s on mixed text and 73.6 t/s on repetitive text at depth 0. At 32K depth decode is 24.1 t/s with 1,177 t/s prefill, and at 128K 23.1 t/s with 1,045 t/s prefill. Eight concurrent users get 86.1 t/s total plain and 73.8 t/s with DSpark on mixed text. IQ2_XXS now runs with experts streamed from a Samsung 990 PRO SSD, at 20.8 t/s decode and 1,064 t/s prefill at depth 0.

generation:

54-79 tokens/s

prompt processing (prefill):

1556.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8 Flash-Next at 1,556 tok/s prefill on a Geekom A9 Mega (Ryzen AI Max+ 395, 128 GB LPDDR5X), with decode swinging 54-79 tok/s as draft acceptance varies between 53% and 80%. Setup is Halowin Bubo, a Gufo fork for Windows 11 and Strix Halo, running the UD-Q4_K_XL GGUF at 108.6 GiB across 8 files, with 96 GB carved out to the GPU in BIOS and a fixed 128 GB page file. Context window is 262K. User says it does not slow down over long context; a 20.3M-token session at a 98% cache hit never pushed the live window past 73%. Load takes 24-26 s. The engine is three weeks old, Windows-only by design, and the user is not affiliated with its authors.

generation:

32.6 tokens/s

prompt processing (prefill):

3581.0 tokens/s

quant:

bf16 (BF16)

kv:

bf16

User reports Qwen3.6-35B-A3B at 3,581 t/s prefill and 32.6 t/s decode at 2k context on a Strix Halo 128GB APU, comparing stock vLLM against the same vLLM with the libstrix HIP kernel plugin. Setup is vLLM with bf16 weights and bf16 KV cache, single stream on one APU; the plugin installs via pip and only activates on ROCm gfx1151. At 64k the same run goes from 260 to 2,617 t/s prefill and 12.8 to 26.9 t/s decode; at 128k with int8 KV it reaches 1,751 t/s prefill and 26 t/s decode, and a 256k prompt prefills in 241 s versus 7,568 s stock. KLD against stock is 0.001-0.006 with 97-99% top-1 agreement, and the user notes decode gains are absent where the replaced kernels are not the bottleneck.

generation:

23.3 tokens/s

prompt processing (prefill):

1077.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

generation:

50.0 tokens/s

prompt processing (prefill):

1400.0 tokens/s

User reports Qwen3.8 Flash-Next on a Strix Halo 128GB (Framework Desktop) at 1400 t/s prefill and about 50 t/s generation, which they call good enough for their usage. Setup is Gufo, serving the model alongside embedding and STT/TTS models; the 50 t/s is described as an average. The user is weighing selling their RTX 5090 to scale out for three concurrent users with 2-3 sessions each at 200k context, and is leaning toward buying a second Strix Halo.

generation:

57.2 tokens/s

prompt processing (prefill):

1637.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User benchmarks Qwen3.8-Flash-Next on a Strix Halo 128GB under Gufo v0.3.0 on Windows 11, reporting 57.16 t/s MTP decode and 1,637 t/s prefill at 8,192 tokens of context. Setup is Gufo with UD-Q4_K_XL weights plus an MTP Q8_0 draft, greedy decoding, two repetitions, one process at a time. MTP decode holds 87.71 t/s at 262,144 tokens with 1,464 t/s prefill; without MTP, autoregressive decode is 33.52 t/s at 8K and 27.35 t/s at 256K. MTP acceptance is 69% at 8K and 100% at longer contexts. The user compares v0.3.0 against v0.2.0 and notes the release build showed heavy disk activity.

generation:

18.9 tokens/s

prompt processing (prefill):

740.0 tokens/s

User reports GLM-5.3 Flash at 18.9 tok/s plain decode on a Strix Halo 128GB box (Ryzen AI Max+ 395, 8060S gfx1151). Setup is a fork of gufo that splits prefill between the GPU and the XDNA2 NPU and stacks speculative decoding on top. With DFlash2 speculation on code the model reaches 29.7 tok/s. Prefill peaks at 740 tok/s at 8k, 17.5% over GPU only. The same fork also runs Flint, Qwen3.6-35B-A3B, Qwen3.8 Flash-Next and Qwen3.8-27B, with best-case decode of 73, 159, 118 and 62 tok/s respectively.

generation:

46-53 tokens/s

prompt processing (prefill):

770.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next 180B-A6B at 46-53 tok/s generation and ~770 tok/s prompt processing on a Strix Halo mini PC with 128 GB unified memory. Setup is llama.cpp and later the Strata engine, with UD-IQ4_XS GGUF weights in 3 shards at 94 GB, plus MTP speculative decoding and a vision projector. The same box ran Qwen3.6-35B-A3B at 76.5 tok/s, and that model did 11.7 tok/s CPU-only on the user's Ryzen 7 6800U laptop. The 46-53 tok/s figure is a range, so no single generation rate is recorded.

generation:

52.7 tokens/s

prompt processing (prefill):

1250.0 tokens/s

quant:

EXL3 4.05 bpw (GGUF)

kv:

q8_0

User reports Qwen3.8-Flash-Next at 52.7 t/s generation with multi-token prediction and about 1,250 t/s prompt processing on a 6K-token prompt, on a Strix Halo 128GB box running Windows. Setup is Rulith Inference 0.6.3 with turboderp's EXL3 4.05 bpw quant repacked as GGUF, q8_0 KV cache, 64 GB VRAM setting with overflow experts in system RAM, 512K context and up to 8 parallel chats. Generation is 28.8 t/s without MTP. The EXL3 build is 104 GB against 111 GB for UD-Q4_K_XL, with mean KLD 0.022 versus 0.024 and code KLD 0.0044 versus 0.0086; chat and multilingual KLD are a tie. The 39 GB n-gram embedding table is read from disk by rows and never goes to the GPU. Prompt processing is a few percent slower than the UD-IQ4_XS build.

generation:

98.0 tokens/s

prompt processing (prefill):

3700.0 tokens/s

quant:

int4

User reports Flint, a 40B MoE with about 2B active parameters, decoding at ~98 tok/s on a Ryzen AI Max+ 395 with 128 GB of unified memory. Setup is the halogen server (0.18.14-flint-preview image) with int4 weights, at 1-2k context. Decode falls to ~67-70 tok/s at 32k context, where prefill is ~2,450 tok/s against ~3,700 tok/s on a 2k prompt. Four concurrent requests total ~190 tok/s, and follow-up turns resume from cache with ~0.4 s to first token at 32k.

generation:

31.0 tokens/s

quant:

Q4_K_M (GGUF)

User compares Nemotron 3.5 Super VL 120B-A12B at 31 t/s against Qwen 3.8 27B at 15 t/s on an AMD Strix Halo 128GB. Setup is LM Studio on Windows 11, with the Nemotron in Q4_K_M and the Qwen in Q8_0. Both models were asked to write a single canvas HTML animation explaining noise cancelling headphones; the user says the Nemotron result is clearly better and hits above its weight class. Nemotron thought for 7 minutes, Qwen for 20.

prompt processing (prefill):

806.0 tokens/s

User reports NPU-assisted prefill on a Strix Halo (Ryzen AI MAX+ 395) box using a fork of Gufo, with the GPU and NPU splitting each projection. Prefill reaches 806 t/s on Qwen3.8-27B, 3,276 t/s on Qwen3.6-35B-A3B and 2,125 t/s on Qwen3.8 Flash-Next. Decode with speculative decoding (MTP, DFlash2, or an auto mode that picks per request) hits up to about 160 tok/s on code and JSON with the 35B-A3B. Quality checked against llama.cpp on CPU showed no measurable perplexity change on the 27B or 35B. The GPU fabric clock must be held at max or the GPU and NPU together can hang the machine; Linux only, gfx1151 only.

generation:

102.0 tokens/s

quant:

QAT UD-Q4_K_XL (GGUF)

User reports Gemma 4 26B-A4B QAT at 102/102/127 tok/s (Swedish prose / English prose / JSON) on a Strix Halo 128GB box, about 45% faster than the regular UD-Q4_K_M at 70/71/90 tok/s. Setup is llama.cpp Vulkan/RADV build 11235 with QAT UD-Q4_K_XL and the matching QAT MTP drafter, single request, -ctxcp 0, using 17.7 GB versus 20.9 GB for Q4_K_M. Lower K-quants (Q3_K_XL, IQ3_S) gave no speed gain. Nemotron 3.5 Lightning 30B-A3B and Qwen3.6-35B-A3B made factual errors in 50-80% of summaries versus about 10% for Gemma. Strata v0.1.44.1 with Qwen3.8-Flash-Next IQ3_S decodes at 50.9 tok/s shallow context and passed 256k needle tests up to 242k tokens with prefill flat near 1,200 tok/s.

generation:

97.7 tokens/s

prompt processing (prefill):

1115.3 tokens/s

quant:

Q4_K_S (GGUF)

kv:

f16

User reports 97.73 t/s generation and 1115.3 t/s prompt processing. Source evidence (verbatim): Source: https://www.soothill.io/blog/2026/08/03/llamacpp-vulkan-vs-rocm-strix-halo/ # llama.cpp: Vulkan vs ROCm on Strix Halo Host: EVO-X3, 128GB integrated-memory Strix Halo desktop. llama.cpp Vulkan b10216 and ROCm b10085. 6 threads pinned, 32,768 context, flash attention on, batch/ubatch 2048/512, K/V cache f16, mmap off. Qwen3-Coder-30B-A3B-Instruct Q4_K_S (17.46 GB), prompt 512, generation 128, single stream: Vulkan 5 runs prompt 1,115.30 tok/s, generation 97.73 tok/s. ROCm 50 runs prompt 1,344.65, generation 73.65. Qwen3.5-122B-A10B Q4_K_XL (77.03 GB), ROCm: pp512 339.9, pp4096 523.0, generation 21.3 tok/s. Vulkan: pp512 285.0, pp4096 374.3, generation 22.9 tok/s. No speculative decoding stated.

prompt processing (prefill):

2013.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8 Flash-Next prefill at 2,013 tok/s on a Ryzen AI MAX+ 395 (Strix Halo 128GB) using the Gufo engine with UD-IQ4_XS quant. Setup uses GPU + XDNA2 NPU prefill fork, with MTP and caches on for the served 42k cold run at 1,836 tok/s. Prefill scales from 1,837 tok/s GPU-only to 2,013 tok/s with NPU at 8,192 tokens, and from 1,788 to 1,949 tok/s at 32,768 tokens.

generation:

51.3 tokens/s

prompt processing (prefill):

2372.0 tokens/s

User reports Qwen3.8-Flash-Next at 2,372 tok/s prefill on a 16K prompt and 51.3 tok/s decode at 8K context on Strix Halo 128GB. Setup is the hipfire 0.4.2.1 engine with MTP speculative decoding on, the XDNA2 NPU handling trunk input projections for two prefill banks alongside the iGPU. Prefill falls to 1,865 tok/s at 128K cold and 1,589 tok/s at 256K cold; decode is 41.8 tok/s at 32K, 42.1 at 128K and 35.6 at 256K. Numbers were measured on a 7.3-rc7 kernel with the IOMMU PerfOpt series, and the user expects lower prefill on stock kernels.

generation:

60.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 60 tok/s on a Strix Halo 128GB laptop (ASUS ProArt PX13, Ryzen AI Max+ 395, 96 GB set aside for GPU). Setup is Gufo (Thomas9120's Windows fork) with UD-Q4_K_XL quant and Q8_0 MTP draft model, 262144 context, speculative decoding with 7 draft tokens and prompt lookup enabled. Figures come from an 80-turn overnight coding run going from 45K to 262K context, not a controlled benchmark. User notes prompt lookup adds 20-30% on code-edit turns and the 10.16 draft changes add another 20-25%.

generation:

28.1 tokens/s

quant:

NVFP4 (NVFP4)

User reports Nemotron 3.5 Super VL 120B-A12B at 28.1 tok/s on Strix Halo 128GB (Framework Desktop, Ryzen AI Max+ 395, gfx1151) under Linux with ROCm 7.13. Setup is Atlas Inference with NVFP4 weights and a LiLiCorr drafter (gamma 8) at 32K context with prefix caching on. Windows 11 gives 27.5 tok/s with the drafter and 18.4 tok/s without; Linux gives 17.8 tok/s without. Speculative decoding is lossless, byte-identical to plain decoding across all 995 BFCL prompts. BFCL ST-995 scores 88.14 and MLPerf Edge Agentic 0.611 across 1,007 recorded turns on both platforms. Text-only on AMD for now; vision runs on DGX Spark.