Local model performance on your hardware

generation:

49.2 tokens/s

prompt processing (prefill):

937.5 tokens/s

quant:

Q5_K_XL (GGUF)

User benchmarks Qwen3.6-35B-A3B at 49.19 t/s generation and 937.47 t/s prompt processing on two AMD MI50 16GB cards. Setup is llama.cpp build b11325 with Vulkan, Q5_K_XL GGUF, flash attention on, 99 GPU layers, and each card's power limit lowered to 145 watts. The table also covers eight other models on the same dual-MI50 rig, with generation rates from 12.05 to 60.57 t/s. A third MI50 is on hand but not yet benchmarked.

generation:

48.0 tokens/s

quant:

Q4_0 (GGUF)

User reports Qwen3.8-27B at 48 tok/s generation on two AMD Instinct MI50 16GB cards, up from 19.4 tok/s with stock llama.cpp. Setup is patched llama.cpp (b11235) on ROCm 7.14 with Q4_0 weights, tensor split plus MTP speculative decoding with 3 draft tokens, 85 W per card. The 48 tok/s is the patched result; the same setup without the gfx906 kernel patches gave 40-44 tok/s, and a vLLM gfx906 fork with GPTQ-Int4 and TP=2 reached 34.7 tok/s.

generation:

30.1 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

User reports Qwen3.8-27B Q8_0 at about 30 tok/s decode on 3x AMD Instinct MI50 16GB (gfx906) with 262,144 configured context. Setup is llama.cpp with HIP/ROCm 6.2.4, RCCL, tensor split mode, F16 KV cache, flash attention on, and about 43.7 GiB aggregate VRAM use. At 216k occupied context decode was 18.24 tok/s with 187.75 tok/s prompt throughput. With HIP graphs enabled, a tiny request after a ~236k stress run stayed at 8.74 tok/s until the server was restarted; disabling graphs (GGML_CUDA_DISABLE_GRAPHS=1) cost about 3% at short context but restored ~30 tok/s immediately after the long-context run.

generation:

16.9 tokens/s

quant:

IQ4 (GGUF)

kv:

Q8

User reports Qwen3.8 Flash-Next at 16.9 t/s on a single MI50 16GB with the gfx906-16gb-expert-pool llama.cpp fork. Setup uses the moe-expert-pool branch with IQ4 GGUF weights, 128K context and Q8 KV cache; the expert cache profile of 66 admits 144 pools at roughly 69% hit rate and peaks at 15.24 GiB VRAM. A stock CPU MoE path without the expert cache reached 11.76 t/s, and up to 19.8 t/s was observed once warm.