Local model performance on your hardware

reported speed:

66.0 tokens/s generation

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 66.04 tokens/s on an AMD Radeon RX 9060 XT 16 GB with 32 GB of system RAM. Setup is llama.cpp (Vulkan backend) with UD-Q4_K_XL GGUF weights, q8_0 KV cache, 40k context, and MTP speculative decoding drafting up to 3 tokens per step. The 35B MoE model does not fit in 16 GB of VRAM, so the expert weights of the first 20 layers run on the CPU. Run on a Ryzen 7 5800X3D desktop under SteamOS, single slot, with flash attention and prefix cache reuse enabled.

reported speed:

30.0 tokens/s generation

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User asks whether Qwen3.8 Flash-Next can run on an RX 9060 XT 16GB with 32GB DDR5-6000. They currently run Swift-1.5-Qwen3.8 at IQ3_S at 30 t/s on the same machine.

reported speed:

50.0 tokens/s generation · 850.0 tokens/s prompt processing

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 850 t/s prefill and 50 t/s generation with MTP at temperature 0 on an AMD RX 9060 XT 16GB, running Qwen3.8-27B-GSQ-RCO IQ3_S in a custom inference engine. MTP is used for the generation figure; the user notes MTP with temperature above 0 is not developed yet. User asks for ideas to systematically test the engine, having already run KL divergence against BF12 on CPU, bit correctness checks, needle-in-a-haystack at 25/50/75/90% key positions with distractors (also against llama.cpp), and HumanEval (92/93 passed so far).

reported speed:

33.5 tokens/s generation · 1210.3 tokens/s prompt processing

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen 2.5-Coder 14B Instruct Q4_K_M at 33.51 t/s generation and 1210.29 t/s prompt processing on an AMD RX 9060 XT 16GB via Vulkan. Setup is llama.cpp build c4ae9a88f8 with -ngl 99 and -fa 1; the same model on ROCm reaches 30.79 t/s generation and 1211.32 t/s prompt processing. A Qwen 3.5 9B Q4_K_M run reaches 50.15 t/s generation and 1904.16 t/s prompt processing on Vulkan, and a 14B quant sweep gives about 32, 28 and 20 tok/s for Q4_K_M, Q5_K_M and Q8_0.

reported speed:

36.0 tokens/s generation · 950.0 tokens/s prompt processing

quant:

Q6_K (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Ornith-1.5-9B at about 36 t/s generation and about 950 t/s prompt eval on an AMD RX 9060 XT 16GB with llama.cpp. Throughput drops to about 25 t/s generation and about 500 t/s prompt eval during long tasks. The run lasted about 3.5 hours on a coding agent task. The user contrasts this with Qwen 3.8 27B, which was frustratingly slow on the same hardware.