Local model performance on your hardware

generation:

43.0 tokens/s

prompt processing (prefill):

1539.0 tokens/s

quant:

AWQ W4A16 (AWQ)

User reports Qwen3.8-Flash-Next at 43 tok/s decode and 1,539 tok/s prefill on a Strix Halo 128GB at a 5.6k prompt. Setup is a vLLM fork with the libstrix plugin, AWQ W4A16 weights, 16k prefill chunks and MTP 2, single stream. The 102 GB n-gram table is streamed from disk and prefetched into the page cache. At 26.9k and 100.5k prompts decode is 43 and 40 tok/s with prefill 1,617 and 1,572 tok/s. Before this work the same setup did 684-806 tok/s prefill and about 18 tok/s decode. KLD against stock is 0.008-0.019 with 97-98% top-1 agreement, perplexity within 0.6%, and GSM8K 93 vs 92.

generation:

35.0 tokens/s

prompt processing (prefill):

1888.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 35.0 t/s short-context decode and 1888 t/s prefill on a Strix Halo 128GB (Ryzen AI Max+ 395) box. Setup is Gufo with UD-Q4_K_XL GGUF weights (103.69 GiB across four shards), GPU only, no MTP, FCLK locked at 2000 MHz, ROCm 7.2.3. At 16K context decode is 34.1 t/s. Stock llama.cpp ROCm HIP on the same machine gave around 568 t/s prefill and 23-24 t/s decode. The model is the ~125B MoE with ~6B active plus n-gram embedding and MTP modules, fitting in the 128 GB pool with about 117 GiB GTT reported.

generation:

24.5 tokens/s

prompt processing (prefill):

1228.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 24.5 t/s decode and 1,228 t/s prefill on a single Strix Halo with their Gufo fork. Setup is Gufo with the IQ1_M GGUF resident, HTTP with real chat prompts, greedy decoding and thinking off; DSpark speculation reaches 30.7 t/s on mixed text and 73.6 t/s on repetitive text at depth 0. At 32K depth decode is 24.1 t/s with 1,177 t/s prefill, and at 128K 23.1 t/s with 1,045 t/s prefill. Eight concurrent users get 86.1 t/s total plain and 73.8 t/s with DSpark on mixed text. IQ2_XXS now runs with experts streamed from a Samsung 990 PRO SSD, at 20.8 t/s decode and 1,064 t/s prefill at depth 0.

generation:

54-79 tokens/s

prompt processing (prefill):

1556.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8 Flash-Next at 1,556 tok/s prefill on a Geekom A9 Mega (Ryzen AI Max+ 395, 128 GB LPDDR5X), with decode swinging 54-79 tok/s as draft acceptance varies between 53% and 80%. Setup is Halowin Bubo, a Gufo fork for Windows 11 and Strix Halo, running the UD-Q4_K_XL GGUF at 108.6 GiB across 8 files, with 96 GB carved out to the GPU in BIOS and a fixed 128 GB page file. Context window is 262K. User says it does not slow down over long context; a 20.3M-token session at a 98% cache hit never pushed the live window past 73%. Load takes 24-26 s. The engine is three weeks old, Windows-only by design, and the user is not affiliated with its authors.

generation:

32.6 tokens/s

prompt processing (prefill):

3581.0 tokens/s

quant:

bf16 (BF16)

kv:

bf16

User reports Qwen3.6-35B-A3B at 3,581 t/s prefill and 32.6 t/s decode at 2k context on a Strix Halo 128GB APU, comparing stock vLLM against the same vLLM with the libstrix HIP kernel plugin. Setup is vLLM with bf16 weights and bf16 KV cache, single stream on one APU; the plugin installs via pip and only activates on ROCm gfx1151. At 64k the same run goes from 260 to 2,617 t/s prefill and 12.8 to 26.9 t/s decode; at 128k with int8 KV it reaches 1,751 t/s prefill and 26 t/s decode, and a 256k prompt prefills in 241 s versus 7,568 s stock. KLD against stock is 0.001-0.006 with 97-99% top-1 agreement, and the user notes decode gains are absent where the replaced kernels are not the bottleneck.

generation:

23.3 tokens/s

prompt processing (prefill):

1077.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

generation:

50.0 tokens/s

prompt processing (prefill):

1400.0 tokens/s

User reports Qwen3.8 Flash-Next on a Strix Halo 128GB (Framework Desktop) at 1400 t/s prefill and about 50 t/s generation, which they call good enough for their usage. Setup is Gufo, serving the model alongside embedding and STT/TTS models; the 50 t/s is described as an average. The user is weighing selling their RTX 5090 to scale out for three concurrent users with 2-3 sessions each at 200k context, and is leaning toward buying a second Strix Halo.

generation:

57.2 tokens/s

prompt processing (prefill):

1637.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User benchmarks Qwen3.8-Flash-Next on a Strix Halo 128GB under Gufo v0.3.0 on Windows 11, reporting 57.16 t/s MTP decode and 1,637 t/s prefill at 8,192 tokens of context. Setup is Gufo with UD-Q4_K_XL weights plus an MTP Q8_0 draft, greedy decoding, two repetitions, one process at a time. MTP decode holds 87.71 t/s at 262,144 tokens with 1,464 t/s prefill; without MTP, autoregressive decode is 33.52 t/s at 8K and 27.35 t/s at 256K. MTP acceptance is 69% at 8K and 100% at longer contexts. The user compares v0.3.0 against v0.2.0 and notes the release build showed heavy disk activity.

generation:

42-46 tokens/s

prompt processing (prefill):

966-1186 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User compares Qwen3.8-Flash-Next and its Swift 1.5 fine-tune across Halogen, gufo and llama.cpp on a Strix Halo 128GB machine, all with MTP. The gufo run with Unsloth's UD-Q4_K_XL decodes at 42-46 t/s with 966-1,186 t/s prefill at a 32k prompt and 64k context. Setup is gufo with the 111.3 GB UD-Q4_K_XL GGUF and Unsloth's shared Q8_0 MTP head; TTFT was 20-27 s and server start 25 s. Halogen's own .hgn build decodes faster at 51-60 t/s, and llama.cpp trails at 20-23 t/s decode and 298-320 t/s prefill. The user suggests gufo with the Swift requant for coding, which finished the 11 Pi tasks in 24 min against 33 min for Unsloth's file on gufo.

generation:

18.9 tokens/s

prompt processing (prefill):

740.0 tokens/s

User reports GLM-5.3 Flash at 18.9 tok/s plain decode on a Strix Halo 128GB box (Ryzen AI Max+ 395, 8060S gfx1151). Setup is a fork of gufo that splits prefill between the GPU and the XDNA2 NPU and stacks speculative decoding on top. With DFlash2 speculation on code the model reaches 29.7 tok/s. Prefill peaks at 740 tok/s at 8k, 17.5% over GPU only. The same fork also runs Flint, Qwen3.6-35B-A3B, Qwen3.8 Flash-Next and Qwen3.8-27B, with best-case decode of 73, 159, 118 and 62 tok/s respectively.

generation:

46-53 tokens/s

prompt processing (prefill):

770.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next 180B-A6B at 46-53 tok/s generation and ~770 tok/s prompt processing on a Strix Halo mini PC with 128 GB unified memory. Setup is llama.cpp and later the Strata engine, with UD-IQ4_XS GGUF weights in 3 shards at 94 GB, plus MTP speculative decoding and a vision projector. The same box ran Qwen3.6-35B-A3B at 76.5 tok/s, and that model did 11.7 tok/s CPU-only on the user's Ryzen 7 6800U laptop. The 46-53 tok/s figure is a range, so no single generation rate is recorded.

generation:

52.7 tokens/s

prompt processing (prefill):

1250.0 tokens/s

quant:

EXL3 4.05 bpw (GGUF)

kv:

q8_0

User reports Qwen3.8-Flash-Next at 52.7 t/s generation with multi-token prediction and about 1,250 t/s prompt processing on a 6K-token prompt, on a Strix Halo 128GB box running Windows. Setup is Rulith Inference 0.6.3 with turboderp's EXL3 4.05 bpw quant repacked as GGUF, q8_0 KV cache, 64 GB VRAM setting with overflow experts in system RAM, 512K context and up to 8 parallel chats. Generation is 28.8 t/s without MTP. The EXL3 build is 104 GB against 111 GB for UD-Q4_K_XL, with mean KLD 0.022 versus 0.024 and code KLD 0.0044 versus 0.0086; chat and multilingual KLD are a tie. The 39 GB n-gram embedding table is read from disk by rows and never goes to the GPU. Prompt processing is a few percent slower than the UD-IQ4_XS build.

generation:

98.0 tokens/s

prompt processing (prefill):

3700.0 tokens/s

quant:

int4

User reports Flint, a 40B MoE with about 2B active parameters, decoding at ~98 tok/s on a Ryzen AI Max+ 395 with 128 GB of unified memory. Setup is the halogen server (0.18.14-flint-preview image) with int4 weights, at 1-2k context. Decode falls to ~67-70 tok/s at 32k context, where prefill is ~2,450 tok/s against ~3,700 tok/s on a 2k prompt. Four concurrent requests total ~190 tok/s, and follow-up turns resume from cache with ~0.4 s to first token at 32k.

generation:

31.0 tokens/s

quant:

Q4_K_M (GGUF)

User compares Nemotron 3.5 Super VL 120B-A12B at 31 t/s against Qwen 3.8 27B at 15 t/s on an AMD Strix Halo 128GB. Setup is LM Studio on Windows 11, with the Nemotron in Q4_K_M and the Qwen in Q8_0. Both models were asked to write a single canvas HTML animation explaining noise cancelling headphones; the user says the Nemotron result is clearly better and hits above its weight class. Nemotron thought for 7 minutes, Qwen for 20.

prompt processing (prefill):

806.0 tokens/s

User reports NPU-assisted prefill on a Strix Halo (Ryzen AI MAX+ 395) box using a fork of Gufo, with the GPU and NPU splitting each projection. Prefill reaches 806 t/s on Qwen3.8-27B, 3,276 t/s on Qwen3.6-35B-A3B and 2,125 t/s on Qwen3.8 Flash-Next. Decode with speculative decoding (MTP, DFlash2, or an auto mode that picks per request) hits up to about 160 tok/s on code and JSON with the 35B-A3B. Quality checked against llama.cpp on CPU showed no measurable perplexity change on the 27B or 35B. The GPU fabric clock must be held at max or the GPU and NPU together can hang the machine; Linux only, gfx1151 only.

generation:

50-60 tokens/s

quant:

IQ4 (GGUF)

User asks which inference engine people prefer on Windows, Strata or Rulith, on a Strix Halo. They run Rulith with an IQ4 quant of Qwen3.8 Flash-Next Uncensored and get around 50 to 60 t/s. They have not tried Strata yet and are asking what others use on Windows.

generation:

52-54 tokens/s

quant:

QAT Q4_0 (GGUF)

User benchmarks 12 local models on a Ryzen AI MAX+ 395 with Radeon 8060S and 128 GB unified memory, scoring four tasks (reasoning, management analysis, coding, MCP tool use) out of 100. Setup is LM Studio with GGUF files at a configured 131,072-token context; speeds are observed generation rates, not end-to-end latency. Gemma 4 26B-A4B QAT Q4_0 and Qwen3.8 27B Q6_K tie at 90/100, at 52-54 t/s and 15-17 t/s respectively; Qwen3.8 was the only model that checked an existing file before overwriting it. GPT-OSS 120B MXFP4 scored 87 at 29-30 t/s with partial GPU offloading. Each model was run once per task and scores are subjective rubric evaluations.

generation:

102.0 tokens/s

quant:

QAT UD-Q4_K_XL (GGUF)

User reports Gemma 4 26B-A4B QAT at 102/102/127 tok/s (Swedish prose / English prose / JSON) on a Strix Halo 128GB box, about 45% faster than the regular UD-Q4_K_M at 70/71/90 tok/s. Setup is llama.cpp Vulkan/RADV build 11235 with QAT UD-Q4_K_XL and the matching QAT MTP drafter, single request, -ctxcp 0, using 17.7 GB versus 20.9 GB for Q4_K_M. Lower K-quants (Q3_K_XL, IQ3_S) gave no speed gain. Nemotron 3.5 Lightning 30B-A3B and Qwen3.6-35B-A3B made factual errors in 50-80% of summaries versus about 10% for Gemma. Strata v0.1.44.1 with Qwen3.8-Flash-Next IQ3_S decodes at 50.9 tok/s shallow context and passed 256k needle tests up to 242k tokens with prefill flat near 1,200 tok/s.

generation:

97.7 tokens/s

prompt processing (prefill):

1115.3 tokens/s

quant:

Q4_K_S (GGUF)

kv:

f16

User reports 97.73 t/s generation and 1115.3 t/s prompt processing. Source evidence (verbatim): Source: https://www.soothill.io/blog/2026/08/03/llamacpp-vulkan-vs-rocm-strix-halo/ # llama.cpp: Vulkan vs ROCm on Strix Halo Host: EVO-X3, 128GB integrated-memory Strix Halo desktop. llama.cpp Vulkan b10216 and ROCm b10085. 6 threads pinned, 32,768 context, flash attention on, batch/ubatch 2048/512, K/V cache f16, mmap off. Qwen3-Coder-30B-A3B-Instruct Q4_K_S (17.46 GB), prompt 512, generation 128, single stream: Vulkan 5 runs prompt 1,115.30 tok/s, generation 97.73 tok/s. ROCm 50 runs prompt 1,344.65, generation 73.65. Qwen3.5-122B-A10B Q4_K_XL (77.03 GB), ROCm: pp512 339.9, pp4096 523.0, generation 21.3 tok/s. Vulkan: pp512 285.0, pp4096 374.3, generation 22.9 tok/s. No speculative decoding stated.

prompt processing (prefill):

2013.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8 Flash-Next prefill at 2,013 tok/s on a Ryzen AI MAX+ 395 (Strix Halo 128GB) using the Gufo engine with UD-IQ4_XS quant. Setup uses GPU + XDNA2 NPU prefill fork, with MTP and caches on for the served 42k cold run at 1,836 tok/s. Prefill scales from 1,837 tok/s GPU-only to 2,013 tok/s with NPU at 8,192 tokens, and from 1,788 to 1,949 tok/s at 32,768 tokens.