- generation:
- 71.1 tokens/s
- prompt processing (prefill):
- 1195.0 tokens/s
- quant:
- ROCmFP4 STRIX hybrid embF16/headQ6 (GGUF)
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 at 71.1 t/s generation and 1195 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is the charlie12345/rocmfp4-llama fork with the ROCmFP4 STRIX hybrid embF16/headQ6 quant, run through Vulkan0 with -ngl 999 -fa 1 -ub 1024 -r 3 and no MTP.
The 71.1 t/s is TG128 and the 1195 t/s is PP2048, separate synthetic tests rather than one request; the same campaign's ROCm0 arm reached 63.4 t/s TG128 and 1271 t/s PP2048.
- prompt processing (prefill):
- 148.8 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-27B UD-Q4_K_XL prefill at 148.8 t/s on an AMD Strix Halo 128GB machine.
Setup is llama.cpp via lemonade ROCm 7 gfx1151 with f16 K and V cache, flash attention on, batch 2048 and 16 CPU workers, single Radeon 8060S with full layer offload.
The figure is the 131072-token input cell of a PP-only sweep with two measured repetitions per cell and depth 0.
- prompt processing (prefill):
- 32.1 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-27B UD-Q4_K_XL at 32.1 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with f16 K and V cache, flash attention on, batch 2048 and 16 CPU workers, one Radeon 8060S with full layer offload.
The figure is the 131072-token prefill cell of a PP-only sweep with depth 0 and n_gen 0; the source's exact mean is 32.145328 t/s, and no decode rate is supplied.
- generation:
- 64.0 tokens/s
- prompt processing (prefill):
- 1005.3 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 64 t/s generation and 1005.3 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Q4_K_M, f16 K and V cache, flash attention on, batch 2048, microbatch 512, and 16 CPU workers.
The figures are the quiet-mode baseline of a five-cell campaign, measured as separate PP512 and TG128 tests, not a 512-in/128-out chat.
- generation:
- 13.0 tokens/s
- prompt processing (prefill):
- 382.5 tokens/s
- quant:
- Q4_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 3 27B IT QAT at 13 t/s generation and 382.5 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with Q4_0 and f16 K and V cache, flash attention on, batch 2048, one GPU with full layer offload.
The stored figures are the quiet baseline cell, PP 382.490359 ± 1.929762 and TG 12.962264 ± 0.046976 t/s; other campaign cells reach up to 437.558764 t/s PP and 13.436544 t/s TG.
- generation:
- 35.9 tokens/s
- prompt processing (prefill):
- 125.7 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- flash attention:
- on
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 35.9 t/s generation and 125.7 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is EngramHalo with UD-Q4_K_XL, q8_0 target K/V cache, Flash Attention, 16384 tokens per slot and eight slots, with draft-mtp plus ngram-mod speculative decoding.
The 35.9 t/s is the server's native TG value for the first one-user request, 71 input and 256 output tokens; client aggregate throughput at one user is 33.2 t/s.
- generation:
- 21.4 tokens/s
- prompt processing (prefill):
- 122.9 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
- kv:
- Q8
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21.4 t/s generation and 122.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is EngramHalo with UD-IQ4_XS, q8_0 target K/V cache, Flash Attention, eight slots of 20480 tokens, and one participating ROCm0 GPU.
The headline is the first one-user request: 71 input tokens and 256 output tokens, with 0.6 s TTFT. The 21.4 t/s is the displayed native TG mean, not client aggregate throughput, which was 20.4 t/s.
- generation:
- 24.9 tokens/s
- prompt processing (prefill):
- 396.8 tokens/s
- quant:
- UD-Q2_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 24.9 t/s generation and 396.8 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q2_K_XL, f16 K/V cache, Flash Attention, batch 2048, microbatch 512 and 16 CPU threads, with 99 GPU layers on the Radeon 8060S iGPU.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one chat run, and are means of two repetitions.
- generation:
- 24.3 tokens/s
- prompt processing (prefill):
- 395.0 tokens/s
- quant:
- UD-Q2_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 24.3 t/s generation and 395 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q2_K_XL, f16 KV cache, Flash Attention, 99 GPU layers on the Radeon 8060S iGPU, batch 2048 and microbatch 512.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not a single chat run; the source estimates about 3% run noise.
- generation:
- 20.9 tokens/s
- prompt processing (prefill):
- 419.6 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- Q8
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 20.9 t/s generation and 419.6 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL and q8_0 KV cache, batch 2048, microbatch 512, 16 CPU threads, Flash Attention enabled, 99 GPU layers on one Radeon 8060S.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one 512-input/128-output chat; the source estimates about 3% run noise.
- generation:
- 21.2 tokens/s
- prompt processing (prefill):
- 405.9 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21.2 t/s generation and 405.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL, f16 KV cache, Flash Attention, 99 GPU layers and 16 CPU threads.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one chat run; the headline cell used microbatch 512.
- generation:
- 21.0 tokens/s
- prompt processing (prefill):
- 411.9 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21 t/s generation and 411.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL, f16 KV cache, Flash Attention, batch 2048, microbatch 512, and 16 CPU threads.
The figures come from separate synthetic PP512 and TG128 tests at depth zero, not a single chat run; the source estimates about 3% run noise.
- generation:
- 50.1 tokens/s
- prompt processing (prefill):
- 563.2 tokens/s
- quant:
- Q6_K (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Ornith-1.5-35B-A3B at 50.1 t/s generation and 563.2 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp b11414 with Vulkan, Q6_K GGUF, one integrated Radeon 8060S, and 32,768 existing KV tokens.
The two figures come from separate depth-32768 tests: prompt processing adds 4096 tokens to the existing cache, and generation covers 128 tokens at that starting depth, not one end-to-end request.
- generation:
- 86.6 tokens/s
- quant:
- BF16 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks SGLang 0.5.17, vLLM 0.26.0 and llama.cpp b10333 on a 128GB GMKtec EVO-X3 with Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151). On Qwen3.5-0.8B BF16, custom vLLM reached 86.59 tok/s at one client and 311.59 tok/s aggregate at eight clients, beating SGLang and llama.cpp at every concurrency.
Setup used ROCm 7.14, 4096-token API limit, deterministic 598-638 token prompts with 64 output tokens, prefix caching disabled. vLLM won all 20 matched concurrency points against SGLang across Qwen3.5-0.8B, Qwen3-8B, Qwen3-Coder-30B-A3B and a shared-prefix workload.
For Qwen3.5-122B-A10B, custom llama.cpp with UD-Q4_K_XL reached 13.57 tok/s at one client and official llama.cpp 17.88 tok/s aggregate at two clients. DeepSeek V4 Flash on a specialised Lucebox ROCmFP3 service reached 23.70 tok/s median.
- generation:
- 54.1 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 54.1 t/s on a Beelink GTR9 Pro with Ryzen AI Max+ 395 and 128 GB unified memory.
Setup is Gufo with UD-Q4_K_XL and a shared Q8_0 MTP drafter at 8K context, one session, 80% draft acceptance.
Gufo loads crash the machine with a data fabric sync flood 4 of 7 times, at random points 9-40 GiB into load, never after a successful load; one load ran 46 hours. Vulkan llama.cpp runs the same model at ~45 t/s structured and ~32 t/s prose on the old config, and gpt-oss-120b at 54.9 t/s. Qwen3.8-27B ran at 12.3 t/s and Flash-Next without MTP at 26.8 t/s.
- generation:
- 114.6 tokens/s
- prompt processing (prefill):
- 3443.0 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DiffusionGemma 26B-A4B at up to 114.6 t/s on Strix Halo 128GB, compared against GB10.
Setup is Gufo with NVFP4 weights, 2K context, single request. Prefill reaches 3443 t/s at 2K and 1379 t/s at 121K. Time to first token is 5.5 s at 2K and 104.7 s at 121K.
User compares to a llama.cpp Q8_0 run on the same hardware, where prefill is 978 t/s and TTFT 18.8 s at 2K. Against GB10 on SPEED-Bench prompts, Strix Halo is slower at 2K (114.6 vs 148.9 t/s) but faster at 32K (80.4 vs 77.9 t/s). User notes remaining headroom: FP8 denoising step at 134 ms against a 95 ms floor, expert weights streaming at 180-200 GB/s of 242 GB/s, dense products at 72% of matrix-unit floor, and 32K prefill at 40% of its floor.
- generation:
- 6.3 tokens/s
- prompt processing (prefill):
- 515.7 tokens/s
- quant:
- UD-Q8_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B It at 6.35 tokens/s generation and 515.74 tokens/s prefill on a Strix Halo 128GB unified-memory system (Ryzen AI MAX+ 395, Radeon 8060S).
Setup is the Gufo Gemma development fork (franzmoca/gufo feat/gemma, not stock Gufo 0.12) with UD-Q8_K_XL GGUF weights, one warmed HTTP single-session prose request, greedy temperature 0, thinking off, autoregressive with no speculation. Actual prompt_n 2038, predicted_n 128.
Rates are engine-stage figures from the harness (prefill_tokens*1000/prefill_ms, completion_tokens*1000/decode_ms), not prompt/TTFT or whole-request throughput, and were not independently recomputed or reproduced; one sample, no SD. Explicit KV precision and Flash Attention setting unknown.