Local model performance on your hardware

reported speed:

34.6 tokens/s generation · 653.9 tokens/s prompt processing

quant:

IQ4_NL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Gemma 4 12B (IQ4_NL, 6.24 GiB) on an AMD RX 6700 XT under llama.cpp, comparing the ROCm and Vulkan backends at 8192-token prefill and 512-token generation with q8_0 KV cache and flash-attention on. ROCm averages 653.9 t/s prefill and 34.60 t/s decode over 3 runs; Vulkan averages 354.4 t/s prefill and 40.92 t/s decode. ROCm is 84.5% faster on prefill but 15.4% slower on decode, giving a net wall-clock win of about 23% for a full 8192-prefill plus 512-generate cycle, with a crossover near 1760 prompt tokens. ROCm required two workarounds on gfx1031: building for gfx1030 with HSA_OVERRIDE_GFX_VERSION=10.3.0, and patching a flash-attention assert in fattn-common.cuh. A separate TOP_K sampler gap on ROCm is noted as under investigation.

reported speed:

61.0 tokens/s generation · 851.0 tokens/s prompt processing

quant:

Q4_K_M (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3-8B Q4_K_M at 851 t/s prompt and 61 t/s generation on an RX 6700 XT 12 GB. Setup is llama.cpp with the AMD Flash Attention kernel, KV cache f16, measured with pp512 / tg128. The project also lists Qwen3.6-35B-A3B Q4_K_S at 475 t/s prompt and 29 t/s generation with --n-cpu-moe 24, and gpt-oss-20B Q4_K_M at 1305 t/s prompt and 94 t/s generation with all experts in VRAM.

reported speed:

18.0 tokens/s generation · 150.0 tokens/s prompt processing

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 18 t/s generation and 150 t/s prompt processing on a 2007 Dell Precision T5400 with dual RX 6700 XT / RX 6700 (22GB total VRAM). Setup is llama.cpp with UD-IQ3_XXS quant at 144k context, using MTP q4_0 speculative decoding, on 24GB DDR2 and dual Xeon X5460. User compares five systems and argues older dual-Xeon dual-GPU setups beat a 2025 HP Omen with RTX 5070 in context length and speed, concluding DDR5 is not worth the cost for agentic tasks.