Local model performance on your hardware

generation:

30.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 30 t/s generation on an RX 6700 XT 12GB with a Qwen3.6 35B-A3B Tiel-Coder model at Q4_K_XL, using llama.cpp with the Vulkan backend on Windows 11. Setup uses 100,000 context, Q8_0 KV cache, flash attention, 28 MoE layers offloaded to CPU, 6 threads, and MTP speculative decoding with draft-n-max 3. User is asking for advice to reach 40 t/s without changing the quant, and notes 6 threads was best out of 4, 8, and 12.

generation:

61.0 tokens/s

prompt processing (prefill):

851.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-8B Q4_K_M at 61 t/s generation and 851 t/s prompt on an RX 6700 XT 12 GB. Setup is the bundled llama.cpp (0.85.5) with the AMD Flash Attention kernel, KV cache f16, measured with pp512 / tg128. The project's headline claim is that the same card goes from 0.6-2.6 t/s under stock llama.cpp to ~61 t/s with ToshLLM. The page also lists Qwen3.6-35B-A3B Q4_K_S at 475 t/s prompt / 29 t/s generation with --n-cpu-moe 24, and community rows including a 122B-A10B MoE across four W6800X GPUs at 226 t/s prompt / 26 t/s generation.

generation:

34.6 tokens/s

prompt processing (prefill):

653.9 tokens/s

quant:

IQ4_NL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Gemma 4 12B (IQ4_NL, 6.24 GiB) on an AMD RX 6700 XT under llama.cpp, comparing the ROCm and Vulkan backends at 8192-token prefill and 512-token generation with q8_0 KV cache and flash-attention on. ROCm averages 653.9 t/s prefill and 34.60 t/s decode over 3 runs; Vulkan averages 354.4 t/s prefill and 40.92 t/s decode. ROCm is 84.5% faster on prefill but 15.4% slower on decode, giving a net wall-clock win of about 23% for a full 8192-prefill plus 512-generate cycle, with a crossover near 1760 prompt tokens. ROCm required two workarounds on gfx1031: building for gfx1030 with HSA_OVERRIDE_GFX_VERSION=10.3.0, and patching a flash-attention assert in fattn-common.cuh. A separate TOP_K sampler gap on ROCm is noted as under investigation.

generation:

18.0 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 18 t/s generation and 150 t/s prompt processing on a 2007 Dell Precision T5400 with dual RX 6700 XT / RX 6700 (22GB total VRAM). Setup is llama.cpp with UD-IQ3_XXS quant at 144k context, using MTP q4_0 speculative decoding, on 24GB DDR2 and dual Xeon X5460. User compares five systems and argues older dual-Xeon dual-GPU setups beat a 2025 HP Omen with RTX 5070 in context length and speed, concluding DDR5 is not worth the cost for agentic tasks.