Local model performance on your hardware

generation:

133.0 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Qwen3-8B Q8_0 with DSpark speculative decoding in llama.cpp on an RX 7900 GRE 16GB, reaching 133 t/s on ROCm versus 59 t/s without speculation, and 115 t/s versus 62 t/s on Vulkan. Setup is llama.cpp with Q8_0 weights, greedy decoding, one user and 32 prompts per setting. DFlash gave 1.5-1.9x, a 0.6B draft model about 1.3x, and n-gram lookup did nothing. The best number of guessed tokens is 3 on Vulkan and 7 on ROCm, and prompt processing is about 8% slower with DSpark loaded. Only 5-6 of 16 answers matched normal decoding word for word.

generation:

51.9 tokens/s

prompt processing (prefill):

322.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports a 36% improvement in generation speed with a dual GPU setup of an RX 7900 GRE 16GB and an RX 480 8GB. The user tested medgemma-27b-it-UD-Q6_K_XL, Qwen3.8-27B-Q6_K, Qwen3.8-27B-OBLITERATED-Q5_K_M, and Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M, with and without Flash Attention.

generation:

11.8 tokens/s

prompt processing (prefill):

87.5 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3.8-27B, identified as Qwen3.5 in llama.cpp, using llama.cpp RPC across multiple GPUs. Setup is an RX 7900 GRE 16GB as the main host, with RPC combinations including RX 7900 GRE + GTX 1080Ti and GTX 1080Ti + two P102-100, in Q4_K_M and Q6_K quants. Results vary by config; the best generation is about 12.53 t/s with Q6_K on RX 7900 GRE + GTX 1080Ti.