AMD Strix Halo 128GB reports

generation:

71.3 tokens/s

prompt processing (prefill):

399.8 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-30B-A3B-Instruct at 71.3 t/s generation and 399.8 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF, Flash Attention on, Vulkan backend, and ngl=100 on the single integrated Radeon 8060S. The figures are separate llama-bench microbenchmarks: tg128 at 71.29 +/- 0.11 t/s and pp512 at 399.76 +/- 1.41 t/s, with longer runs of tg8192 at 59.91 +/- 1.95 t/s and pp8192 at 315.36 +/- 0.33 t/s.

generation:

73.0 tokens/s

prompt processing (prefill):

391.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-30B-A3B-Instruct at 73 t/s generation and 391.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF, Flash Attention off, Vulkan backend, and ngl=100 on the integrated Radeon 8060S. The source-native rates are separate llama-bench microbenchmarks: pp512 at 391.54 +/- 1.50 t/s and tg128 at 72.99 +/- 0.31 t/s, with longer tests at pp8192 284.71 t/s and tg8192 46.60 t/s.

generation:

9.8 tokens/s

prompt processing (prefill):

97.9 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3 at 9.8 t/s generation and 97.9 t/s prompt processing on two AMD Strix Halo nodes with 128GB each. Setup is the staged GLM path over OdinLink USB4/TB5 RDMA with Q4_K, TP=2 across two Radeon 8060S iGPUs, fixed 4,096 prompt tokens and 300 generated tokens at prefill batch 256. The source gives exact rates of 97.89 t/s prefill and 9.77 t/s decode from a single diagnostic run, with no TTFT reported and no single-node scaling claimed.

generation:

19.9 tokens/s

prompt processing (prefill):

270.1 tokens/s

quant:

Q4_K experts / mixed F16-Q8 (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 19.9 t/s generation and 270.1 t/s prompt processing on two AMD Strix Halo nodes with 128 GB RAM each, linked by OdinLink USB4/TB5 RDMA. Setup is the Antirez Q4_K experts build with F16 HC/compressor/indexer and Q8 attention/shared/output, tensor parallel across two Radeon 8060S iGPUs with a balanced 50/50 expert split. The run used a fixed 2,048-token prompt followed by 300 generated tokens, with ordinary decode and no speculative decoding; the source gives 19.89 t/s and 270.10 t/s exactly.

generation:

10.1 tokens/s

prompt processing (prefill):

103.4 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash Q4_K at 10.1 t/s generation and 103.4 t/s prompt processing on two AMD Strix Halo nodes, each with 128 GB installed memory. Setup is a custom DS4 fork's staged GLM TP=2 ROCm path with a 4,608-token context allocation, 4,096-token prompt and 300-token decode at prefill batch 256, no speculative draft or MTP. The 4K point uses a cross-disciplinary prompt and is one observation; the source waived repeated timing, so no spread or confidence interval is established.

generation:

22.3 tokens/s

prompt processing (prefill):

214.2 tokens/s

quant:

PQ2_0 (GGUF)

kv:

Q4

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Bonsai 2 at 22.3 t/s generation and 214.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is PrismML-Eng/llama.cpp with PQ2_0 weights, Q4_0 K/V cache, full GPU offload, flash attention, one slot and 262144 context on a single Ryzen AI MAX+ 395 APU with 128 GB unified memory. The figure is the idle-peer fixed-prefix run with 524 input and 256 output tokens; the source gives 22.25989819108211 t/s and 214.23028774725407 t/s.

generation:

49.5 tokens/s

prompt processing (prefill):

358.5 tokens/s

quant:

W4B HGN + quality overlay

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 49.5 t/s generation and 358.5 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is Halogen 0.9.1 with a W4B HGN quant plus quality overlay, 262144 context per request, MTP and prompt lookup enabled, on one Ryzen AI MAX+ 395 APU with 128 GB unified memory. The 49.5 t/s and 358.5 t/s come from a single idle-peer fixed-code repeat with 524 input and 256 output tokens; the same run's client-side generation was 50.974291±1.201111 t/s.

generation:

36.8 tokens/s

prompt processing (prefill):

284.5 tokens/s

quant:

Q4_0_ROCMFP4_FAST (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 36.8 t/s generation and 284.5 t/s prefill on an AMD Strix Halo 128GB box. Setup is the ROCmFPX llama.cpp fork with Q4_0_ROCMFP4_FAST, q8_0 K and V cache, flash attention on, one parallel slot, 131072 context, and native MTP with max 4 draft tokens. The 36.8 t/s is the median of four prompt categories at temp 0, single stream with 600-token outputs; the 284.5 t/s prefill is a separate 11,626-token prompt measurement.

generation:

12.2 tokens/s

prompt processing (prefill):

291.6 tokens/s

quant:

Q4_K_XL (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 12.2 t/s generation and 291.6 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 434ddbb with the Q4_K_XL quant, Vulkan backend, single Radeon 8060S GPU, and a 1GB BIOS carve plus 120GB GTT region. The figures are separate synthetic PP512 and TG128 tests, not a 512-input/128-output request; the source means are 291.62 +/- 28.60 t/s and 12.15 +/- 0.02 t/s over 5 repeats.

generation:

18.6 tokens/s

prompt processing (prefill):

140.3 tokens/s

quant:

IQ3_XXS (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash IQ3_XXS at 18.6 t/s generation and 140.3 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 434ddbb with Vulkan on a single Radeon 8060S, 120GB GTT region, and a 97.05GiB tensor size. The figures are separate PP512 and TG128 synthetic tests, not a 512-input/128-output client request; the source gives 18.56 +/- 0.01 t/s and 140.30 +/- 4.35 t/s over 3 repeats.

generation:

13.2 tokens/s

prompt processing (prefill):

370.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Muse Glimmer 30B at 13.2 t/s generation and 370.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 11146 with Q4_K_M, F16 K/V cache, ngl 999, FA1, batch 2048 and ubatch 512, one Vulkan device. The headline pair is pass 1 pp512/tg128 at prepared depth 0, r10; tg128 at prepared depth 8192 measured 13.0 t/s.

generation:

80.4 tokens/s

prompt processing (prefill):

1399.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Laguna XS 2.1 at 80.4 t/s generation and 1399 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 11146 with the Q4_K_M quant, F16 K/V cache, ngl 999, FA1, batch 2048 and ubatch 512, on a single Radeon 8060S Vulkan device. The headline pair is pass 1 pp512/tg128 at prepared depth 0, r10; the source notes these are synthetic tests, not a 512-input/128-output chat request.

generation:

66.8 tokens/s

prompt processing (prefill):

1450.3 tokens/s

quant:

Q4_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Nemotron 3.5 Lightning 30B-A3B at 66.8 t/s generation and 1450.3 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 11146 with Q4_0, F16 K/V cache, ngl 999, FA1, batch 2048 and ubatch 512 on a single Radeon 8060S Vulkan device. The headline pair is pass 1 pp512 and tg128 at prepared depth 0, r10; the depth 8192 tg128 run used r3.

generation:

54.2 tokens/s

prompt processing (prefill):

1049.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B-MTP Q8_0 at 54.2 t/s generation and 1049.7 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 10603 with Vulkan RADV/Mesa 26.1.7, -ngl 999 -b 2048 -ub 512 -fa on, plain decoding with no speculation, one Radeon 8060S GPU. The headline pairs same-backend Vulkan pp512 1049.70 ±7.40 t/s with tg128 54.22 ±0.09 t/s; the reported SD 0.00 is rounded, not proof of zero variability.

generation:

7.8 tokens/s

prompt processing (prefill):

278.8 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Q8_0 at 7.8 t/s generation and 278.8 t/s prompt processing on an AMD Ryzen AI Max+ 395 / Radeon 8060S with 128 GiB unified RAM. Setup is llama.cpp build 10603 with Vulkan RADV/Mesa 26.1.7, -ngl 999 -b 2048 -ub 512 -fa on, F16 K/V, one GPU, 5 repetitions. The headline pairs same-backend Vulkan pp512 278.80 ±9.40 t/s with tg128 7.77 ±0.00 t/s; the reported SD of 0.00 is rounded, not proof of zero variability. TG128 is 128 sequential synchronized random-token decode work units, not delivered text, and the PP sizes are timed synthetic work, not maximum context capacity.

generation:

85.1 tokens/s

prompt processing (prefill):

1917.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 85.1 t/s generation and 1917.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q4_K_M GGUF, ROCm/HIP 7.2.4, -ngl 999 and flash attention, on one enumerated gfx1151 device. The figures are llama-bench PP512 and TG128 means, not client-visible tokens; the source also lists Ollama prompt-eval 880.53 and eval 82.08, which the user excludes from the native metrics.

generation:

73.0 tokens/s

prompt processing (prefill):

961.2 tokens/s

quant:

MXFP4 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss-20b at 73 t/s generation and 961.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with MXFP4 and f16 K/V cache, one ROCm device, ngl 999 and flash attention, no speculative drafter. The prompt figure is noisy, 961.24 ±390.50 t/s, which the author attributes to GPU contention from live services; generation is 73.01 ±0.10 t/s and described as more reliable.

generation:

73.7 tokens/s

prompt processing (prefill):

843.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-30B-A3B at 73.7 t/s generation and 843.3 t/s prompt processing on an AMD Strix Halo 128GB box. Setup is llama.cpp with a Q4_K_M GGUF, full GPU offload (ngl 999) and flash attention on ROCm/HIP 7.2.4, one ROCm device. The headline is the d0 arm, PP843.25 ±5.39 and TG73.74 ±0.17 tok/s; at depth 4096 it drops to PP209.17 and TG47.48 tok/s, where the depth is pre-existing KV state rather than a timed prefill.

generation:

12.1 tokens/s

prompt processing (prefill):

334.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B-MTP Q4_K_M at 12.1 t/s generation and 334 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q4_K_M and F16 KV cache, one ROCm/Vulkan device, flash attention on and mmap off, without speculative decoding. The figures are native llama-bench PP512/TG128 means over 3 repetitions at depth 0; the raw logs give 333.95±7.08 t/s and 12.05±0.03 t/s.

generation:

21.5 tokens/s

prompt processing (prefill):

484.0 tokens/s

quant:

BF16 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 21.5 t/s generation and 484 t/s prompt processing on an AMD Strix Halo 128GB. Setup is llama.cpp with BF16 weights and ROCm flash attention, one GPU device with positive layer offload. The 21.5 t/s and 484 t/s come from separate fixed-work tests, PP512 and TG128, not one chat request; a separate shape 2 run at depth 0 gives 23.71 t/s generation.