- generation:
- 6.5 tokens/s
- prompt processing (prefill):
- 60.8 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
- kv:
- F16
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 6.5 t/s generation and 60.8 t/s prompt processing on a Mac mini M4 with 24 GB unified memory.
Setup is llama.cpp build 9544 with the UD-IQ4_XS GGUF, run via llama-bench with -p 512 -n 128, F16 K/V cache and flash attention AUTO.
The source's native figures are PP512 = 60.78 +/- 1.60 tokens/s and TG128 = 6.47 +/- 0.03 tokens/s, separate prompt and generation microbenchmarks rather than one request; TG128 is depth-0 generation, not decode after a 512-token prompt.
- generation:
- 36.8 tokens/s
- prompt processing (prefill):
- 284.5 tokens/s
- quant:
- Q4_0_ROCMFP4_FAST (GGUF)
- kv:
- Q8
- flash attention:
- on
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 36.8 t/s generation and 284.5 t/s prefill on an AMD Strix Halo 128GB box.
Setup is the ROCmFPX llama.cpp fork with Q4_0_ROCMFP4_FAST, q8_0 K and V cache, flash attention on, one parallel slot, 131072 context, and native MTP with max 4 draft tokens.
The 36.8 t/s is the median of four prompt categories at temp 0, single stream with 600-token outputs; the 284.5 t/s prefill is a separate 11,626-token prompt measurement.
- generation:
- 12.2 tokens/s
- prompt processing (prefill):
- 291.6 tokens/s
- quant:
- Q4_K_XL (GGUF)
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 12.2 t/s generation and 291.6 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp build 434ddbb with the Q4_K_XL quant, Vulkan backend, single Radeon 8060S GPU, and a 1GB BIOS carve plus 120GB GTT region.
The figures are separate synthetic PP512 and TG128 tests, not a 512-input/128-output request; the source means are 291.62 +/- 28.60 t/s and 12.15 +/- 0.02 t/s over 5 repeats.
- generation:
- 7.8 tokens/s
- prompt processing (prefill):
- 278.8 tokens/s
- quant:
- Q8_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Q8_0 at 7.8 t/s generation and 278.8 t/s prompt processing on an AMD Ryzen AI Max+ 395 / Radeon 8060S with 128 GiB unified RAM.
Setup is llama.cpp build 10603 with Vulkan RADV/Mesa 26.1.7, -ngl 999 -b 2048 -ub 512 -fa on, F16 K/V, one GPU, 5 repetitions.
The headline pairs same-backend Vulkan pp512 278.80 ±9.40 t/s with tg128 7.77 ±0.00 t/s; the reported SD of 0.00 is rounded, not proof of zero variability. TG128 is 128 sequential synchronized random-token decode work units, not delivered text, and the PP sizes are timed synthetic work, not maximum context capacity.
- generation:
- 13.0 tokens/s
- prompt processing (prefill):
- 125.6 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 13 t/s generation and 125.6 t/s prompt processing on an M4 Pro with 64 GB unified memory.
Setup is dasLLAMA version 16 with UD-IQ4_XS weights and f16 KV cache, one GPU, 640 positions allocated.
The figures are native PP512 and TG128 means over five repetitions, not one 512-input/128-output request.
- generation:
- 13.1 tokens/s
- prompt processing (prefill):
- 126.2 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 13.1 t/s generation and 126.2 t/s prompt processing on an M4 Pro with 64 GB unified memory.
Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU.
The figures are native PP512 and TG128 means over five repetitions, not a single chat request; the session allocates 640 positions.
- generation:
- 12.8 tokens/s
- prompt processing (prefill):
- 126.1 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 12.8 t/s generation and 126.1 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory.
Setup is llama.cpp commit 6fdd0ac89 with the UD-IQ4_XS GGUF, Metal backend, -ngl 99, 10 CPU threads, and K/V cache in f16.
The figures are native llama-bench means over five repetitions: PP512 at 126.094155 t/s and TG128 at 12.841348 t/s, each starting from cleared state at depth 0, with model load and warmup excluded.
- generation:
- 11.5 tokens/s
- prompt processing (prefill):
- 126.3 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 11.5 t/s generation and 126.3 t/s prompt processing on an M4 Pro 64GB.
Setup is llama.cpp commit 6fdd0ac89 with Q4_K_M, Metal backend, -ngl 99, 10 CPU threads, and K/V f16.
The native figures are PP512 and TG128 from cleared state at depth 0, not a single 512-input/128-output request.
- prompt processing (prefill):
- 509.0 tokens/s
- quant:
- Q4_0-H32 W4A4 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 509 t/s prompt processing on one AMD Strix Halo APU/iGPU, named as Ryzen AI Max+ 395 with Radeon 8060S.
Setup is the 1bit engine lean ROCm build with Q4_0-H32 W4A4 and a 32-point Walsh-Hadamard-rotated activation path, run through native llama-bench pp512 at microbatch 512.
The figure is prompt-processing only; generation is left blank because the card's roughly 11 tok/s and a later site's roughly 13 tok/s lack a matched native tg128 measurement. Context capacity, iteration count and memory are unreported.
- generation:
- 35.9 tokens/s
- prompt processing (prefill):
- 125.7 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- flash attention:
- on
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 35.9 t/s generation and 125.7 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is EngramHalo with UD-Q4_K_XL, q8_0 target K/V cache, Flash Attention, 16384 tokens per slot and eight slots, with draft-mtp plus ngram-mod speculative decoding.
The 35.9 t/s is the server's native TG value for the first one-user request, 71 input and 256 output tokens; client aggregate throughput at one user is 33.2 t/s.
- generation:
- 21.4 tokens/s
- prompt processing (prefill):
- 122.9 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
- kv:
- Q8
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21.4 t/s generation and 122.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is EngramHalo with UD-IQ4_XS, q8_0 target K/V cache, Flash Attention, eight slots of 20480 tokens, and one participating ROCm0 GPU.
The headline is the first one-user request: 71 input tokens and 256 output tokens, with 0.6 s TTFT. The 21.4 t/s is the displayed native TG mean, not client aggregate throughput, which was 20.4 t/s.
- generation:
- 24.9 tokens/s
- prompt processing (prefill):
- 396.8 tokens/s
- quant:
- UD-Q2_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 24.9 t/s generation and 396.8 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q2_K_XL, f16 K/V cache, Flash Attention, batch 2048, microbatch 512 and 16 CPU threads, with 99 GPU layers on the Radeon 8060S iGPU.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one chat run, and are means of two repetitions.
- generation:
- 24.3 tokens/s
- prompt processing (prefill):
- 395.0 tokens/s
- quant:
- UD-Q2_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 24.3 t/s generation and 395 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q2_K_XL, f16 KV cache, Flash Attention, 99 GPU layers on the Radeon 8060S iGPU, batch 2048 and microbatch 512.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not a single chat run; the source estimates about 3% run noise.
- generation:
- 20.9 tokens/s
- prompt processing (prefill):
- 419.6 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- Q8
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 20.9 t/s generation and 419.6 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL and q8_0 KV cache, batch 2048, microbatch 512, 16 CPU threads, Flash Attention enabled, 99 GPU layers on one Radeon 8060S.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one 512-input/128-output chat; the source estimates about 3% run noise.
- generation:
- 21.2 tokens/s
- prompt processing (prefill):
- 405.9 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21.2 t/s generation and 405.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL, f16 KV cache, Flash Attention, 99 GPU layers and 16 CPU threads.
The figures are separate synthetic PP512 and TG128 tests at depth zero, not one chat run; the headline cell used microbatch 512.
- generation:
- 21.0 tokens/s
- prompt processing (prefill):
- 411.9 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 21 t/s generation and 411.9 t/s prompt processing on an AMD Strix Halo 128GB machine.
Setup is llama.cpp with UD-Q4_K_XL, f16 KV cache, Flash Attention, batch 2048, microbatch 512, and 16 CPU threads.
The figures come from separate synthetic PP512 and TG128 tests at depth zero, not a single chat run; the source estimates about 3% run noise.
- generation:
- 56.9 tokens/s
- prompt processing (prefill):
- 1196.5 tokens/s
- quant:
- W4B + default quality overlay
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 56.9 t/s generation and 1196.5 t/s prompt processing on an AMD Radeon 8060S / gfx1151 host with 128 GiB unified memory.
Setup is Halogen 0.14.2 with W4B plus default quality overlay, native MTP, one KV slot and 262144 configured positions on one render device.
The headline is a warm cached-prefix result at 64000 cached depth with 2048 new tokens (66048 total input) and 512 outputs; the preceding 45.17 s prefix refresh and model load are excluded, and the source-normalized rate for this cell is 56.75318 t/s.
- generation:
- 26.0 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 26 t/s on an Apple M5 Max with 36 GB of unified memory.
Setup is LM Studio with MLX 4-bit weights; KV cache and speculative decoding are not established.
The figure is a single-stream decode rate, a median of three requests, not engine-native timing or prompt processing.
- generation:
- 25.3 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 25.3 t/s on an Apple M5 Max with 36 GB unified memory.
Setup is oMLX 0.7.0 with MLX 4-bit weights and an SSD cache directory enabled; speculative decoding is not established.
The figure is a single-stream decode rate from a median of three requests, not engine-native timing or end-to-end throughput, and the source warns differences below roughly 5% can be run variation.
- generation:
- 25.8 tokens/s
- quant:
- 4-bit (MLX)
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 25.8 t/s on an M5 Max with 36 GB of unified memory.
Setup is Rapid-MLX 0.16.1 with MLX 4-bit weights and --no-spec-decode, running text prompts.
The figure is a single-stream streaming decode rate, not engine-native timing or end-to-end throughput, and the harness requests a 512-token cap at temperature 0.