Local model performance on your hardware

reported speed:

46.0 tokens/s generation · 1400.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 1,400 t/s prefill and 46 t/s decode on a Strix Halo 128GB laptop at 70 W. Setup is Gufo as the engine, with the model running at xhigh effort. The user notes the model ran on an older Halogen version (0.14.0) and Halogen dropped the connection once, so the last part ran on Gufo. The user compares the local model against Claude Opus 5.5 on a coding task, finding the local model's PR better in tests and edge cases, though Opus was 2 to 10 times faster overall.

reported speed:

50.0 tokens/s generation

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at about 50 tok/s decode with MTP on a Strix Halo 128GB box, even at 116k context. Setup is a forked Gufo engine with the Unsloth UD-IQ4_XS quant (about 89GB) and an MTP sidecar, running in the 120W performance profile. Prefill is 1500+ tok/s from about 3k tokens up, peaking around 1570; on HumanEval prompts decode is about 76 tok/s with MTP versus 28 without. The balanced profile is around 10% less prefill, and very short prompts are slower (about 1350 at 2k) due to a fixed cost per request.

reported speed:

34.0 tokens/s generation

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B at 33.96 t/s generation on Strix Halo 128GB. Setup is Gufo with Q4_K_XL GGUF and DFlash2 speculative decoding using a Q4_K_M draft model. The prompt processing figure of 183.3 t/s is noted as low because of the short prompt. User notes it is slower on Q8_0 but still faster than llama.cpp, and is testing the new AUR package.

reported speed:

1124.0 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next on a Bosgame M5 (Strix Halo 128GB) across three backends. Gufo v0.7.0 with unsloth UD-Q4_K_XL and MTP reaches 54.0 t/s decode on code, 53.6 t/s on JSON, 29.5 t/s on prose, and 32.9 t/s at 19k context, with prefill 1124/1183/1179 t/s at 4k/19k/38k. strix-llama (ROCm) with ISTA GSQ-RCO IQ3_S and MTP gets 37.9/36.0/24.4/26.6 t/s decode and 734/798/794 t/s prefill; mainline llama.cpp (Vulkan) without MTP gets 27.5/27.5/27.3/25.5 t/s decode and 329/328/285 t/s prefill. GPU memory at 128k context is ~85 GiB for Gufo, ~62 GiB for strix-llama, ~57 GiB for mainline. The user notes Gufo is fastest but only accepts unsloth Q4_K_XL, while IQ3_S on strix-llama leaves room for Gemma 4 26B-A4B alongside. MTP did not work on mainline Vulkan. llama-bench on strix-llama with IQ3_S and no MTP gave pp2048 944 t/s (845 at 32k) and tg128 26.4.

reported speed:

40.0 tokens/s generation

quant:

UD_Q4_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 40 t/s average on agentic tasks on Strix Halo 128GB. Setup is Gufo with UD_Q4_XL quant, VRAM set to 96GB. User notes the speed holds at higher context lengths and recommends setting VRAM to 96GB on Windows.

reported speed:

50.8 tokens/s generation · 2384.0 tokens/s prompt processing

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B at 50.8 t/s decode and 2384 t/s prefill on a Strix Halo 128GB using the gufo engine with Q8_K_XL quantization. Setup is gufo with Q8_K_XL GGUF and no KV cache quantization, running on the Ryzen AI Max+ 395 with 128 GB unified memory. The user compares gufo against llama.cpp across multiple quantizations and speculative decoding modes. Gufo achieves higher prefill speeds (up to 2702 t/s on Q6_K_XL) but llama.cpp decodes faster on Q6_K_XL by 3% to 9%. With DFlash2 speculative decoding at 7 draft tokens, gufo reaches 78.6 t/s on Q8_K_XL versus llama.cpp's 51.7 t/s. The user notes that speculative output is not bit-identical to plain greedy output on either engine.

reported speed:

33.6 tokens/s generation · 1343.0 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8 Flash-Next on a ROG Flow Z13 2025 with Strix Halo 128GB, comparing Halogen, Gufo and Rulith at 60W and 93W. Gufo 0.5.0 with UD-Q4_K_XL and MTP Q8_0 draft 7 reaches 1343 tok/s prefill and 33.6 tok/s decode at 200K context at 93W, and 1156 tok/s prefill with 33.0 tok/s decode at 60W. Halogen 0.16.0 is fastest for prefill, hitting 1694 tok/s at 200K and 41.16 tok/s decode at 93W, while Rulith on Windows is fastest at short context with 54.7 tok/s decode at 1K but falls to 32.55 tok/s at 200K. The user notes the comparison is not apples-to-apples because model formats and MTP setups differ between backends.

reported speed:

32.9 tokens/s generation · 1033.0 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next on an AMD Strix Halo 128GB (ASUS ROG Flow Z13 GZ302) at 70 W TDP, comparing several engines. The gufo engine with UD-Q4_K_XL weights reaches 32.9 t/s decode and 1,033 t/s prefill at 64k context (32k prompt), with 74% MTP acceptance and 14/14 retrieval. Setup uses gufo (ROCm 7.2.4) with UD-Q4_K_XL GGUF weights and MTP speculative decoding, run via LlamaStash on Arch Linux. Halogen 0.14.0 with native .hgn weights is fastest at 39.3 t/s decode and 1,045 t/s prefill, but is closed source and Docker-only. gufo is open source and loads 4x faster from cold. At 128k context (64k prompt), gufo gets 32.3 t/s decode and 1,047 t/s prefill; at 256k context (130k prompt), 27.9 t/s decode and 997 t/s prefill. In 10 Aider polyglot Python exercises, gufo passes 10/10 in 36.0 min, versus Halogen 21.5 min and CIRU 24.6 min.

reported speed:

70.2 tokens/s generation

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 70.22 tok/s on a Strix Halo device, reproducing Gufo's published 70.56 tok/s figure. Setup is Gufo 0.4.0 with UD-Q4_K_XL GGUF weights and a DFlash2 Q4_K_M draft model, greedy decoding, thinking off, 128 output tokens, prompt cache off. The 70 tok/s figure comes from a repetitive prompt ("Write the word red exactly 1000 times") where speculative decoding accepts nearly every draft token. On nine ordinary prompts the median is 39.4 tok/s, ranging from 22 to 52. With 8 users the aggregated figure is 122.6 tok/s but wall-clock token delivery is 82 tok/s on the repetitive prompt and 52 on normal prompts. Without the draft model Gufo's docs put it around 12 tok/s. In a head-to-head against halogen 0.13.8 on Qwen3.8 Flash-Next, halogen averaged 43.9 tok/s vs Gufo's 38.2 on normal prompts, and 76.6 vs 63.1 tok/s with 4 users; Gufo was faster at cold prefill (1495 vs 1288 tok/s) and on the repetitive prompt (87.4 vs 56.9 tok/s).

reported speed:

39.4 tokens/s generation

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 39.4 t/s median on normal prompts on a Strix Halo device. Setup is Gufo 0.4.0 with UD-Q4_K_XL GGUF and a DFlash2 Q4_K_M draft model, single user, speculative decoding on. The 70.22 t/s figure comes from a repetitive prompt and is a ceiling; normal prompts range from 22 to 52 t/s. In a head-to-head with halogen 0.13.8 on identical hardware, halogen was about 13% faster for one user and 18% faster with four, while Gufo was 16% faster at prompt processing and much faster on the repetitive prompt.

reported speed:

70.6 tokens/s generation · 656.3 tokens/s prompt processing

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B Q4 at up to 70.56 t/s generation and 656.33 t/s prompt processing on a Strix Halo 128GB. Setup is the Gufo inference engine, a new vertically integrated engine built specifically for Strix Halo hardware. At concurrency 8 the model reaches 123 t/s aggregate. The engine also supports DeepSeek V4 Flash, Qwen3.8 Flash-Next, Qwen3 ASR and TTS, Qwen Image 2.1, and MiniMax H3.