Local model performance on your hardware

generation:

20.2 tokens/s

prompt processing (prefill):

176.9 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Swift-Qwen3.8-27B-Uncensored-MTP at 20.21 t/s generation and 176.86 t/s prompt processing on dual AMD Radeon Instinct MI50 (32GB combined VRAM) under ROCm 10.2, compared against 18.35 t/s and 149.30 t/s under Vulkan Mesa 26.2. Setup is llama.cpp (llama-b11401 for ROCm, llama-b11382 for Vulkan) with Q6_K GGUF weights, both GPUs power limited to 145 watts. ROCm shows +10.14% generation and +18.46% prefill over Vulkan for this model. Across the full nine-model sweep, ROCm gains up to +62.74% on generation but regresses on MoE prefill by up to -15.98%, with high variance (e.g. ±102.56 on Qwen3.5MoE Q4_K prefill).

generation:

66.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3 Coder 30B A3B at 66.11 t/s generation on one AMD Radeon Instinct Mi50 32GB. Setup is llama.cpp build 128d522c (6686) with the ROCm backend, Q4_K_M quant (17.28 GiB), -ngl 99, 112 threads and --numa distribute on a dual Xeon 8480+ host. Other quants on the same card: Q5_0 65.15 t/s, Q6_K 62.49 t/s, Q8_0 64.20 t/s. BF16 (56.89 GiB) required two GPUs and ran 41.41 t/s. With 16K tokens of output the rate falls to about 20 t/s.

generation:

15.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 15 t/s generation and 100-200 t/s prompt processing at 130k context. Setup is ROCm with flash-attn, split-mode layer and fit on. The user notes Vulkan gives similar speed.

generation:

15.1 tokens/s

prompt processing (prefill):

410.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks the gfx906-llama-cpp fork on an MI50 32GB, reaching 410 t/s prefill at PP16384 against 332.5 t/s upstream, 264 t/s on a 120k deep fill against 231.4 t/s upstream, and 15.1 t/s TG against 13.6 t/s upstream. Outputs are bit-identical to upstream. The user attributes the gains to exploring existing llama.cpp PRs.

generation:

15-16 tokens/s

prompt processing (prefill):

105-110 tokens/s

quant:

UD-IQ2_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 15-16 t/s generation and 105-110 t/s prompt processing on 3x AMD MI50 32GB GPUs, 96GB of VRAM in total. Setup is llama-server (llama.cpp) with the UD-IQ2_M quant. The 90.9GB model fits fully in VRAM. The user tested it with a Rubik's cube HTML coding task and is impressed it runs locally.

generation:

13.6 tokens/s

prompt processing (prefill):

379.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a fork optimized for AMD gfx906 (MI50/MI60/Radeon VII) reaching 379.2 t/s prefill versus 332.3 t/s (+14.1%), 252.6 t/s versus 231.1 t/s on a 120k-context fill (+9.3%), and 13.6 t/s versus 13.5 t/s deep-context TG. DFlash acceptance is 0.691.

quant:

Q8_0 (GGUF)

kv:

Q8

mtp (multi-token prediction):

on

User benchmarks MTP KV cache quantization at Q8_0 against no quantization on Qwen3.6-27B-Q8_0 with llama.cpp. Setup uses tensor parallelism on 2xMI50 32GB. Wall time difference is negligible at roughly 0.14s with the quantized draft KV cache. Aggregate accept rate is 0.735-0.741.