AMD Strix Halo 128GB reports

generation:

6.3 tokens/s

prompt processing (prefill):

515.7 tokens/s

quant:

UD-Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B It at 6.35 tokens/s generation and 515.74 tokens/s prefill on a Strix Halo 128GB unified-memory system (Ryzen AI MAX+ 395, Radeon 8060S). Setup is the Gufo Gemma development fork (franzmoca/gufo feat/gemma, not stock Gufo 0.12) with UD-Q8_K_XL GGUF weights, one warmed HTTP single-session prose request, greedy temperature 0, thinking off, autoregressive with no speculation. Actual prompt_n 2038, predicted_n 128. Rates are engine-stage figures from the harness (prefill_tokens*1000/prefill_ms, completion_tokens*1000/decode_ms), not prompt/TTFT or whole-request throughput, and were not independently recomputed or reproduced; one sample, no SD. Explicit KV precision and Flash Attention setting unknown.

generation:

27.2 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 27.16 t/s on Strix Halo 128GB, from a ranked table of 33 models measured on the same hardware. Setup is llama.cpp Vulkan/RADV with UD-IQ4_XS weights; the same model runs 50.69 t/s on the user's Halogen engine with 79.1% draft acceptance. The table's other rows range from 266.00 t/s (Qwen3 0.6B) to 4.90 t/s (Llama 3.1 70B); the user notes dense models above 14B are below usable speed and that the intelligence column is unsourced opinion, not measured data.

prompt processing (prefill):

1821.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next prefill at 1821 tok/s at 16k context on a Strix Halo 128GB (EVO-X2, Ryzen AI Max+ 395). Setup is a fork of Gufo with UD-IQ4_XS quant, offloading some GEMMs to the NPU while the GPU handles the rest. Output verified identical to GPU-only build across 30 long greedy prompts. Prefill improved from ~1700 tok/s with GPU only. Pinning the GPU fabric clock at max was required to avoid hard hangs and also gave a speedup by itself.

generation:

53.2 tokens/s

prompt processing (prefill):

3991.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 53.2 t/s decode and 3991 t/s prefill on Strix Halo 128GB, compared against llama.cpp b11069 at 44.0 t/s decode and 1685 t/s prefill. Setup is the Gufo fork with Unsloth UD-Q4_K_XL GGUF, single-user HTTP benchmark, greedy decoding with thinking off, no MTP. With MTP speculative decoding decode rises to 79.8 t/s, and 254 t/s on repetitive output. At 131K context decode is 38.0 t/s (47.7 with MTP). With 8 concurrent users the 26B serves about 222 t/s aggregate versus 185 on llama.cpp. The 26B QAT quant reaches 81.4 t/s decode and 4131 t/s prefill in gufo bench versus 69.4 and 1546 on llama-bench.

generation:

56.3 tokens/s

prompt processing (prefill):

4045.0 tokens/s

quant:

Q8_0 (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks a new strix-llama.cpp branch (rdna35-perf) on a Strix Halo 128GB box, reporting prompt processing 21-54% faster than Vulkan and 8-174% faster than stock ROCm, with generation tying Vulkan on the dense 27B and 2-3% behind on MoE and 2B at short context. Setup is llama-bench with Q8_0 weights, q8_0 KV cache, flash attention on, ROCm/HIP backend, ub 4096 for the MoE. The headline Qwen3.6 35B-A3B figures are pp4096 4045 t/s and tg64 56.3 t/s at depth 0. Quality checks show KL divergence closer to stock ROCm than Vulkan and unchanged perplexity. The branch is ROCm/HIP only and was tested on three models: Qwen3.6 35B-A3B, Qwen3.8 27B, and Qwen3.5 2B.

generation:

49.1 tokens/s

prompt processing (prefill):

1028.1 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 49.1 t/s generation and 1028.1 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with UD-Q4_K_XL, f16 key/value cache and FlashAttention on, measured at a depth of 16,384 already-prefilled KV tokens. These are separate llama-bench pp512 and tg128 tests, not a fresh 16,384-token ingest; the same ladder gives 52.73 t/s generation at depth 0 and 50.95 t/s at depth 8,192.

generation:

67.7 tokens/s

prompt processing (prefill):

2886.0 tokens/s

quant:

Q8_0

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 2B at 67.7 t/s generation and 2886 t/s prompt processing on an AMD Strix Halo 128GB box with a Radeon 8060S. Setup is llama.cpp on the rdna35-perf branch with Q8_0 weights, q8_0 KV cache, Flash Attention on and microbatch 2048, measured with llama-bench. The two figures are separate benchmark cells at a source-labeled 128k tokens already in the KV cache, not one end-to-end request; the prompt test processes 2048 additional tokens at that depth and is not TTFT.

generation:

39.9 tokens/s

prompt processing (prefill):

1068.0 tokens/s

quant:

Q8_0

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 39.9 t/s generation and 1068 t/s prompt processing on an AMD Strix Halo 128GB box with a Radeon 8060S. Setup is llama.cpp on the rdna35-perf ROCm/HIP branch with Q8_0 weights, q8_0 KV cache, Flash Attention on and microbatch 4096, measured with llama-bench. Both figures are separate benchmark cells at 128k tokens of cache depth, not one end-to-end request; the prompt test processes 4096 additional tokens at that depth, and configured context is unspecified.

generation:

6.7 tokens/s

prompt processing (prefill):

312.0 tokens/s

quant:

Q8_0

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 6.7 t/s generation and 312 t/s prompt processing on an AMD Strix Halo 128GB box with a Radeon 8060S. Setup is llama.cpp on the rdna35-perf ROCm/HIP branch with Q8_0 weights, q8_0 KV cache, Flash Attention on and microbatch 2048, measured with llama-bench. Both figures are separate benchmark cells at 128k tokens already in the KV cache, not one end-to-end request; the prompt test processes 2048 additional tokens at that depth.

generation:

15.6 tokens/s

prompt processing (prefill):

109.6 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 15.6 t/s generation and 109.6 t/s prefill on an AMD Strix Halo 128GB workstation. Setup is llama.cpp with UD-Q4_K_XL, F16 KV cache and flash attention, 40 GPU layers offloaded in a single inference slot with 2048 MiB prompt-cache capacity, configured to 262144 context. CPU-backed per-layer token embeddings make this a hybrid offload rather than a full-GPU test, and actual input/output lengths and cache hits are unreported.

generation:

41.6 tokens/s

prompt processing (prefill):

1698.9 tokens/s

quant:

Q8_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 41.6 t/s generation and 1698.9 t/s prefill on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q8_K_XL, ROCm 7.2.3 HIP backend, Flash Attention enabled, -ngl 99, -b 2048 and -ub 2048, on one Ryzen AI MAX+ 395 with 128 GB unified memory. The headline combines two separate phase tests, pp512 and tg128, not one paired request; the same model's baseline control reports pp512 = 1369.67 and tg128 = 41.61 t/s.

generation:

45.6 tokens/s

prompt processing (prefill):

1339.4 tokens/s

quant:

Q8_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 45.6 t/s generation and 1339.4 t/s prefill on an AMD Strix Halo 128GB system. Setup is llama.cpp with Q8_K_XL and Flash Attention, using a patched ROCm 7.2.3 container with -ngl 99, -fa 1, -mmp 0, -dio 1, -b 2048, -ub 2048. The headline combines separate pp512 and tg128 phase tests, not one paired request; the baseline control reports 1083.55 t/s prefill and 45.82 t/s generation.

generation:

71.1 tokens/s

prompt processing (prefill):

1488.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 71.1 t/s generation and 1488.7 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is Gufo with UD-Q4_K_XL, a 262,144-token configured context, 512-token prefill chunks and MTP ceiling 7 with a shared Q8 MTP head. The run used 2,048 new prompt tokens at depth zero and 512 generated tokens, with 447 drafted and 447 accepted tokens; the exact rates were 1,488.66 PP and 71.08 TG.

generation:

39.9 tokens/s

prompt processing (prefill):

1060.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next UD-Q4_K_XL at 39.9 t/s decode and 1,060 t/s prefill on an ASUS ROG Flow Z13 with AMD Strix Halo, Ryzen AI Max+ 395 / Radeon 8060S and 128 GB unified memory. Setup is Gufo v0.7.1 built with ROCm 10, one server session on native Linux, with a shared Q8_0 MTP sidecar at 68% draft acceptance. Each row is the median of three fresh prompts, source-labelled 32k prompt within a 64k window, at a 70 W power limit with measured power 67-70 W.

prompt processing (prefill):

454.6 tokens/s

quant:

IQ2_XXS + Q2_K, selected Q8 tensors (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash prefill at 454.6 t/s on an AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151) with 128 GB unified memory. Setup is the DS4 Halo ROCm prefill fork with IQ2_XXS + Q2_K and selected Q8 tensors, 4,352-token context allocation and 4,096-token prefill chunks. The figure is a warmed prepared-prefill mean of 454.59 t/s from two samples, 454.52 and 454.65, on a complete position-zero 4,096-token request with generation disabled.

generation:

56.5 tokens/s

prompt processing (prefill):

655.2 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 56.5 t/s generation and 655.2 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine experts at 4.5175 bpw, F16 target and draft attention cache, a matching Q8 Orca MTP drafter at depth 4, one slot and 262144-token configured context. The run reused all but four tokens of a 256000-token prefix, then appended 2048 tokens and generated 512 output tokens; MTP accepted 408 of 408 proposed tokens. Full append-plus-generation wall time was 12.409 seconds, with no repeated 256K result.

generation:

59.2 tokens/s

prompt processing (prefill):

780.1 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 59.2 t/s generation and 780.1 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine routed experts at 4.5175 bpw, F16 target and draft attention cache, a matching Q8 Orca MTP drafter at depth 4, and 262144-token configured context. The headline is one initial observation at 64,000-token prior prefix depth with 2,048 appended tokens and 512 output tokens, not an aggregate; the reversed-order repeat gave 903.82 t/s append and 58.67 t/s generation.

prompt processing (prefill):

1003.0 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 1003 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with the compact indexed-affine expert format at 4.5175 bpw, a 262144-token configured context, 16384 batch and microbatch, F16 target and draft attention cache, and a Q8 Orca MTP drafter at depth 4. The figure is a full-prefix prefill of 256000 tokens from zero cached prefix, not the later 2048-token cached append and not TTFT; one initial observation, no repeats.

prompt processing (prefill):

1251.4 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 1251.4 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine experts at 4.5175 bpw, F16 target and draft attention cache, a Q8 Orca MTP drafter at depth 4, one slot and 262144-token configured context. The figure is a full-prefix prefill of 64,000 prompt tokens from zero cached prefix, not TTFT and not the later 2048-token cached append; a separate reverse-order 64K run measured 1267.40 t/s.

generation:

81.4 tokens/s

prompt processing (prefill):

4131.0 tokens/s

quant:

QAT UD-Q4_K_XL (Q4_0) (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 81.4 t/s generation and 4131 t/s prompt processing on AMD Strix Halo with 128GB unified memory. Setup is the franzmoca/gufo Gemma fork with native HIP kernels for gfx1151, QAT UD-Q4_K_XL (Q4_0), MTP off. The figures are separate native gufo bench synthetic tests (pp2048 and tg128), not a paired chat request or HTTP end-to-end throughput; the author states three repetitions but not the aggregate statistic.