Local model performance on your hardware

generation:

23.0 tokens/s

prompt processing (prefill):

170.0 tokens/s

quant:

IQ1_M/IQ2_XS/IQ2_XXS (IST-DAS 2.00bpw) (GGUF)

User reports DeepSeek-V4.1-Flash at about 23 tok/s decode and ~170 tok/s prefill on a single AMD MI50 32GB. Setup is Strata with the IST-DAS 2.00bpw GGUF set (IQ1_M/IQ2_XS/IQ2_XXS experts), a separate 111 GB Engram table and the DSpark drafter, about 258 GB on disk, with 62 GB RAM and 24 CPU threads. Only ~23.6 GB of the 132 GB of experts fit on the card; the rest runs from a 46 GB RAM store and the SSD, so weights are offloaded. Decode was 15.8 tok/s by default, 16.7 with 24 CPU threads, 19.6 with --experts 5, and 22.6-23 with DSpark on at 0.65 confidence. Prefill was measured on a 7.5K-token prompt; short ~30-token prompts give ~30-40 tok/s. Contexts were tested only up to 8K with one request at a time, and the gfx906 support is a set of local patches, not upstream.

generation:

32.1 tokens/s

quant:

FP16 (safetensors)

User reports Qwen2.5-1.5B-Instruct at 32.14 tok/s single-stream on an AMD Instinct MI50 32GB with vLLM. Setup is vLLM with unquantized FP16 weights on one MI50, one request, no speculative decoding, enforce-eager, ROCm flash attention, max model length 8192, in an unprivileged Proxmox LXC container. Under continuous batching aggregate throughput rises to 63.8 tok/s at 2 streams, 238.9 at 8 and 422.8 at 32. llama.cpp with 4-bit GGUF weights on the same card gave 31.25 tok/s single-stream and stayed near 31 at every concurrency. Qwen2.5-0.5B reached 648.2 tok/s aggregate.

generation:

20.2 tokens/s

prompt processing (prefill):

176.9 tokens/s

quant:

Q6_K (GGUF)

User benchmarks Swift-Qwen3.8-27B-Uncensored-MTP at 20.21 t/s generation and 176.86 t/s prompt processing on dual AMD Radeon Instinct MI50 (32GB combined VRAM) under ROCm 10.2, compared against 18.35 t/s and 149.30 t/s under Vulkan Mesa 26.2. Setup is llama.cpp (llama-b11401 for ROCm, llama-b11382 for Vulkan) with Q6_K GGUF weights, both GPUs power limited to 145 watts. ROCm shows +10.14% generation and +18.46% prefill over Vulkan for this model. Across the full nine-model sweep, ROCm gains up to +62.74% on generation but regresses on MoE prefill by up to -15.98%, with high variance (e.g. ±102.56 on Qwen3.5MoE Q4_K prefill).

generation:

63.0 tokens/s

prompt processing (prefill):

1224.1 tokens/s

quant:

Q4_0 (GGUF)

kv:

Q8_0

User benchmarks Qwen3-30B-A3B-Thinking-2507-Q4_0 on an AMD MI50 using a specialized llama.cpp-gfx906 fork with flash attention and Q8_0 KV cache quantization. Prompt processing reaches 1224.07 t/s at 512 tokens and generation 63.00 t/s at 128 tokens. Without KV quantization, generation is 79.92 t/s but prompt processing drops to 1167.28 t/s at 512 tokens. The fork is optimized for D=128 head dimension models and reports 5-11% prompt processing improvement from flash attention.

generation:

66.1 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3 Coder 30B A3B at 66.11 t/s generation on one AMD Radeon Instinct Mi50 32GB. Setup is llama.cpp build 128d522c (6686) with the ROCm backend, Q4_K_M quant (17.28 GiB), -ngl 99, 112 threads and --numa distribute on a dual Xeon 8480+ host. Other quants on the same card: Q5_0 65.15 t/s, Q6_K 62.49 t/s, Q8_0 64.20 t/s. BF16 (56.89 GiB) required two GPUs and ran 41.41 t/s. With 16K tokens of output the rate falls to about 20 t/s.

prompt processing (prefill):

95.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Gemma 4 26B at 65-68 tok/s on an AMD Instinct MI50 32GB. Setup is Ollama v0.32.14 with Q4_K_M and q8_0 KV cache, flash attention on, all layers offloaded, MTP speculative decoding enabled automatically. The figure is a range; short answers reach ~95 tok/s prompt eval. A QAT Q4_0 build with MTP reaches ~78 tok/s, and Qwen 3.5 35B Q6_K runs at ~52 tok/s with ~541 tok/s prompt eval. Power overdrive to 160W adds another +8-9%.

generation:

108.9 tokens/s

quant:

Q4_K (GGUF)

kv:

FP16

User reports Ornith-1.5-35B-A3B decoding at 108.9 tok/s on an AMD MI50 32GB, 1.9x the llama.cpp baseline of 57.2 tok/s on the same GGUF and GPU. Setup is a custom HIP engine with Q4_K weights and FP16 KV cache, short context of 273-529 tokens, model load 27.6 s, sclk 1725 MHz under load. The engine reproduces llama.cpp's 8-token greedy continuation 8/8 teacher-forced and free-running; the MTP layer is not loaded.

generation:

22.8 tokens/s

quant:

Q8_0 (GGUF)

kv:

Q8_0

User reports Qwen3.8-27B at 22.8 tok/s decode on a single AMD Instinct MI50 32GB, using the MI50-llama.cpp fork. Setup is llama.cpp with Q8_0 weights and Q8_0 KV cache, flash attention on, one slot, 66560 context. Decode holds 22.0 tok/s at 16k and 20.4 tok/s at 64k; MTP with 3 drafts reaches 28.8 tok/s on a 64k reasoning prompt, 51.8 tok/s on short code, 43.2 tok/s on prose, and 24.2 tok/s on a 106.7k prompt. Max context is 114,688 with -ub 512. All figures are single-stream.

generation:

15.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports 15 t/s generation and 100-200 t/s prompt processing at 130k context. Setup is ROCm with flash-attn, split-mode layer and fit on. The user notes Vulkan gives similar speed.

generation:

15.1 tokens/s

prompt processing (prefill):

410.0 tokens/s

User benchmarks the gfx906-llama-cpp fork on an MI50 32GB, reaching 410 t/s prefill at PP16384 against 332.5 t/s upstream, 264 t/s on a 120k deep fill against 231.4 t/s upstream, and 15.1 t/s TG against 13.6 t/s upstream. Outputs are bit-identical to upstream. The user attributes the gains to exploring existing llama.cpp PRs.

generation:

15-16 tokens/s

prompt processing (prefill):

105-110 tokens/s

quant:

UD-IQ2_M

User reports DeepSeek V4 Flash 0731 at 15-16 t/s generation and 105-110 t/s prompt processing on 3x AMD MI50 32GB GPUs, 96GB of VRAM in total. Setup is llama-server (llama.cpp) with the UD-IQ2_M quant. The 90.9GB model fits fully in VRAM. The user tested it with a Rubik's cube HTML coding task and is impressed it runs locally.

generation:

13.6 tokens/s

prompt processing (prefill):

379.2 tokens/s

User reports a fork optimized for AMD gfx906 (MI50/MI60/Radeon VII) reaching 379.2 t/s prefill versus 332.3 t/s (+14.1%), 252.6 t/s versus 231.1 t/s on a 120k-context fill (+9.3%), and 13.6 t/s versus 13.5 t/s deep-context TG. DFlash acceptance is 0.691.

quant:

Q8_0 (GGUF)

kv:

Q8

mtp (multi-token prediction):

on

User benchmarks MTP KV cache quantization at Q8_0 against no quantization on Qwen3.6-27B-Q8_0 with llama.cpp. Setup uses tensor parallelism on 2xMI50 32GB. Wall time difference is negligible at roughly 0.14s with the quantized draft KV cache. Aggregate accept rate is 0.735-0.741.

generation:

9.7 tokens/s

prompt processing (prefill):

264.0 tokens/s

User reports Kimi K2.6 at 264 t/s prompt processing and 9.7 t/s generation on 32x AMD MI50 32GB GPUs.