AMD Strix Halo 128GB reports

generation:

59.0 tokens/s

prompt processing (prefill):

1828.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma-4-E4B at 59 t/s generation and 1828 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp with UD-Q4_K_XL, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate pp512 and tg128 microbenchmarks, not one end-to-end request; the AMDVLK backend gave 491 t/s prompt and 57 t/s generation.

generation:

88.7 tokens/s

prompt processing (prefill):

2278.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Granite-4.1-3B at 88.7 t/s generation and 2278 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp with UD-Q4_K_XL, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the same GPU reached 88.1 t/s generation under AMDVLK and 2323 t/s prompt processing under ROCm, with no ROCm generation figure published.

generation:

109.0 tokens/s

prompt processing (prefill):

3382.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma-4-E2B at 109 t/s generation and 3382 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp with UD-Q4_K_XL, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the AMDVLK backend gave 954 t/s prompt and 99 t/s generation.

generation:

9.2 tokens/s

prompt processing (prefill):

154.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder 30B at 9.2 t/s generation and 154 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quantization, 6 threads and maximum GPU offload, run as llama-bench with -p 512 -n 128. The source gives 9.19 t/s TG and 154 t/s PP as separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and memory use are unreported.

generation:

8.9 tokens/s

prompt processing (prefill):

106.0 tokens/s

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Mistral-8x22b at 8.9 t/s generation and 106 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_M quant, 6 threads, and maximum GPU offload (-ngl 999). The source gives 8.92 t/s and 106 t/s for the separate PP512 and TG128 phases, rounded here to 8.9 t/s; the two are not an end-to-end rate.

generation:

3.0 tokens/s

prompt processing (prefill):

45.0 tokens/s

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Mistral Large 2411 123B at 3 t/s generation and 45 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_M quant, 6 threads and maximum GPU offload. The source gives 2.97 t/s and 45 t/s for the separate PP512 and TG128 phase tests, rounded to 3 t/s and 45 t/s; the two are not an end-to-end rate.

generation:

3.9 tokens/s

prompt processing (prefill):

104.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 3 27B at 3.9 t/s generation and 104 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 weights, 6 threads and maximum GPU offload (-ngl 999). The figures come from separate synthetic llama-bench phase tests, PP512 and TG128, not an end-to-end rate; the source gives rounded rates without raw durations or repeat counts.

generation:

54.0 tokens/s

prompt processing (prefill):

500.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss 120B at 54 t/s generation and 500 t/s prompt processing on an AMD Strix Halo 128GB machine, a Framework Desktop with AMD AI Max 395+ and 128 GB LPDDR5. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, MXFP4 quantization, 6 CPU threads, and maximum GPU offload (-ngl 999). The figures come from separate llama-bench phase tests, a 512-token prompt and 128-token generation, not an end-to-end rate; rates are rounded and context depth, KV precision, and batch settings are unreported.

generation:

77.0 tokens/s

prompt processing (prefill):

1216.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss 20B at 77 t/s generation and 1216 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp Vulkan on the Radeon 8060S iGPU with MXFP4 quantization, 6 threads and maximum GPU offload (ngl 999). The figures come from separate llama-bench phase tests, PP512 and TG128, not an end-to-end rate; rates are rounded and context depth, KV precision and batch settings are unreported.

generation:

20.2 tokens/s

prompt processing (prefill):

225.0 tokens/s

quant:

Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 4 Scout 17B-16E at 20.2 t/s generation and 225 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp Vulkan on the Radeon 8060S iGPU with Q4_K_XL, 6 threads and maximum GPU offload. The figures come from separate llama-bench phase tests, a 512-token prompt and 128-token generation, not an end-to-end rate; rates are rounded and context depth, KV precision and batch settings are unreported.

generation:

5.1 tokens/s

prompt processing (prefill):

78.0 tokens/s

quant:

Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.3 70B at 5.1 t/s generation and 78 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_XL quant, 6 threads and maximum GPU offload (ngl 999). The source gives TG 5.05 t/s and PP 78 t/s from separate llama-bench phase tests at 512-token prompt and 128-token generation, not an end-to-end rate.

generation:

28.0 tokens/s

prompt processing (prefill):

730.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 28 t/s generation and 730 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 quant, 6 threads and maximum GPU offload, run with llama-bench -p 512 -n 128. The 28 t/s and 730 t/s are separate synthetic phase tests, not an end-to-end rate; context depth, KV precision and batch settings are unreported.

generation:

93.0 tokens/s

prompt processing (prefill):

1912.0 tokens/s

quant:

Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 93 t/s generation and 1912 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, Q4_K_XL quant, 6 threads, and -ngl 999 for maximum GPU offload. The figures come from llama-bench with -p 512 -n 128, so prompt and generation are separate synthetic phase tests, not an end-to-end rate; rates are rounded and no raw timings, repetition count, or context depth are reported.

generation:

185.0 tokens/s

prompt processing (prefill):

5879.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Baguettotron 321M at 185 t/s generation and 5879 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Vulkan on the Radeon 8060S iGPU, BF16 weights, 6 threads, and -ngl 999 for maximum GPU offload. The figures come from separate llama-bench phase tests, PP512 and TG128, not an end-to-end rate; the rates are rounded and no raw timings, context depth, or KV cache details are reported.

generation:

51.1 tokens/s

prompt processing (prefill):

173.5 tokens/s

quant:

MXFP4 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss-120b at 51.1 t/s generation and 173.5 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with MXFP4 weights and f16 KV cache, one Radeon 8060S iGPU with 999 layers offloaded, n_batch and n_ubatch 512, flash attention on. The figures are separate synthetic PP512 and TG128 tests at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

generation:

69.0 tokens/s

prompt processing (prefill):

1538.2 tokens/s

quant:

Q6_K_L (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 69 t/s generation and 1538.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with the Q6_K_L GGUF, f16 KV cache, flash attention on, and 999 GPU layers offloaded to the single Radeon 8060S iGPU. The figures are separate synthetic llama-bench PP512 and TG128 runs at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

generation:

69.6 tokens/s

prompt processing (prefill):

812.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-30B-A3B-Instruct at 69.6 t/s generation and 812 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp with Q4_K_M weights, f16 K/V cache, flash attention, and one GPU (Radeon 8060S) with n_gpu_layers=999. The figures are separate synthetic llama-bench PP512 and TG128 tests at n_depth=0, not generation after a 512-token prompt and not HTTP end-to-end throughput; no long-context rate is implied.

generation:

35.0 tokens/s

prompt processing (prefill):

356.8 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 at 35 t/s generation and 356.8 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is the Nathanw1014 Strix Halo llama.cpp v0.6.10 fork with UD-Q4_K_M, f16/f16 KV cache, flash attention on, 16 CPU threads, ngl999 and Vulkan0, at a depth of 131072 tokens. The figures are native llama-bench pp16384/tg256 results, N=3, with MTP disabled and reasoning off; deeper cells gave 238.8/28.8 at 200000 and 167.4/24.9 at 262144.

generation:

65.9 tokens/s

prompt processing (prefill):

401.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-30B-A3B-Instruct at 65.9 t/s generation and 401.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp with the UD-Q4_K_XL GGUF, ROCm 6.4.0, Flash Attention on, and one gfx1151 iGPU at ngl=100. The source gives 65.93 +/- 0.18 t/s for tg128 and 401.48 +/- 2.72 t/s for pp512, with tg8192 at 63.27 t/s; these are separate synthetic microbenchmarks, not a same-request prompt and answer.

generation:

59.8 tokens/s

prompt processing (prefill):

409.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-30B-A3B-Instruct at 59.8 t/s generation and 409.5 t/s prompt processing on an EVO-X2 with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp with the UD-Q4_K_XL GGUF, ROCm 6.4.0, Flash Attention off, and one gfx1151 iGPU at ngl=100. The rates are separate llama-bench microbenchmarks, not a same-request prompt and answer; longer tests in the same arm give pp8192 254.28 t/s and tg8192 30.78 t/s.