AMD Strix Halo 128GB reports

generation:

21.2 tokens/s

quant:

Q4_K_M (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MTP enabled with --spec-type draft-mtp --spec-draft-n-max 3. Baseline without MTP is 11.7 tok/s. Q8_0 was also tested: 7.4 → 18.1 tok/s (2.44×).

generation:

20.6 tokens/s

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.6-27B Dense with MTP on Strix Halo under Windows, at 18.7 t/s for a poem, 19.8 t/s for editing HTML and 23.2 t/s for creating HTML. Setup is llama.cpp with spec-draft-n-max 3.

generation:

15.5 tokens/s

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.6-27B Dense with MTP at 14.5 t/s on a poem task, 14.2 t/s on an edit HTML task, and 17.9 t/s on a create HTML task, on Strix Halo. Setup is llama.cpp on Windows with spec-draft-n-max 6.

generation:

12.6 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.6-27B Dense without MTP at 12.5 t/s on a poem task, 12.6 t/s on an edit HTML task, and 12.6 t/s on a create HTML task on Strix Halo. Setup is llama.cpp on Windows.

quant:

Q8_0 (GGUF)

flash attention:

on

User benchmarks MTP against non-MTP for 27B and 35B-A3B models. The 27B-MTP run shows significant speedup in generation and overall wall time for long-context chat. The 35B-MTP run shows mixed results: faster generation but slower end-to-end due to prefill overhead.

generation:

21.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports multiple models and quants on Strix Halo 128GB with the ROCm backend, plus tests on an RTX 3090 and an RTX 5070. Setup is the Q4_K_M quant for a chat workload.