Local model performance on your hardware

generation:

66.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 66.04 tokens/s on an AMD Radeon RX 9060 XT 16 GB with 32 GB of system RAM. Setup is llama.cpp (Vulkan backend) with UD-Q4_K_XL GGUF weights, q8_0 KV cache, 40k context, and MTP speculative decoding drafting up to 3 tokens per step. The 35B MoE model does not fit in 16 GB of VRAM, so the expert weights of the first 20 layers run on the CPU. Run on a Ryzen 7 5800X3D desktop under SteamOS, single slot, with flash attention and prefix cache reuse enabled.

generation:

33.5 tokens/s

prompt processing (prefill):

1210.3 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen 2.5-Coder 14B Instruct Q4_K_M at 33.51 t/s generation and 1210.29 t/s prompt processing on an AMD RX 9060 XT 16GB via Vulkan. Setup is llama.cpp build c4ae9a88f8 with -ngl 99 and -fa 1; the same model on ROCm reaches 30.79 t/s generation and 1211.32 t/s prompt processing. A Qwen 3.5 9B Q4_K_M run reaches 50.15 t/s generation and 1904.16 t/s prompt processing on Vulkan, and a 14B quant sweep gives about 32, 28 and 20 tok/s for Q4_K_M, Q5_K_M and Q8_0.

generation:

36.0 tokens/s

prompt processing (prefill):

950.0 tokens/s

quant:

Q6_K (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ornith-1.5-9B at about 36 t/s generation and about 950 t/s prompt eval on an AMD RX 9060 XT 16GB with llama.cpp. Throughput drops to about 25 t/s generation and about 500 t/s prompt eval during long tasks. The run lasted about 3.5 hours on a coding agent task. The user contrasts this with Qwen 3.8 27B, which was frustratingly slow on the same hardware.