Local model performance on your hardware

generation:

40.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash reaching a peak of 40 t/s generation on 8x Radeon AI PRO R9700 under llama.cpp, but the run froze after 200-300 tokens. Setup is llama.cpp on 8x R9700 with 256GB system RAM and roughly 400GB total VRAM; the user notes llama.cpp on many GPUs was very slow, possibly due to ROCm. The user also runs MiniMax M2.7 (MXFP4_16, iq4_nl) on 8x R9700 and Qwen3-27b-BF16 on 4x7900xtx under vLLM, and gave up on a REAP GLM-5.1 50% build due to vLLM errors.

generation:

24.9 tokens/s

prompt processing (prefill):

611.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B Q5_K_M on dual AMD Radeon AI PRO R9700 GPUs, reporting 24.85 t/s generation and 611.02 t/s prompt processing at 32768 context on a single GPU. Setup is llama.cpp with ROCm, Q5_K_M GGUF weights, q8_0 KV cache, flash attention, and 2 GPUs with layer split. Dual GPUs roughly doubled long-context prefill (1215.87 t/s at 32768) but did not improve single-stream generation (24.31 t/s). The user also tested Gemma 4 31B, Qwen3.6 35B A3B, and Gemma 4 26B A4B, finding sparse-active models much faster but not directly comparable to dense models.

generation:

36.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~36 tok/s single stream on a single Radeon AI PRO R9700 32GB with 64GB system RAM. Setup is llama.cpp with stew675's RDNA4 patches (r30), AtomicChat Q4_K_M quant, q8_0 KV cache, flash attention, 2 slots x 131k context, -ncmoe 41 and MOE_EXPERT_CACHE_MIB=4096, GPU capped at 220W. Two streams combined reach ~50 tok/s and prompt reading is 526-589 tok/s. The 27B Q5_K_XL is ~1.7x faster per turn and reads prompts 1.6-2x as fast, but scores lower on tool-eval-bench Hard Mode (79.8 vs 93.9).

generation:

21.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 21 t/s on llama.cpp with IQ3_XXS, running on system RAM plus a Radeon AI PRO R9700 32GB. Setup is llama.cpp with IQ3_XXS, model partially offloaded to system RAM (64GB DDR5). User also reports ~60 t/s with the same quant on Strata, and 35 t/s for Qwen3.8-27B Q6 on the R9700. System RAM usage reaches 96% with Flash-Next loaded, preventing simultaneous ComfyUI workloads.

generation:

29.9 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 29.92 t/s on a mixed GPU pool of R9700, RTX 5070 Ti, and RTX 4070 using llama.cpp RPC. Setup is llama.cpp with Q8 quant and 256k context, using MTP speculative decoding. The RPC pool runs each card on its native driver (ROCm for R9700, CUDA for Nvidias). User compares this to Vulkan solo on the R9700 (26.48 t/s) and their daily driver Q6_K_XL at 131k context (59.08 t/s). RPC showed tighter consistency (spread 7.4 vs 16.3). Flash Next saw no improvement.

generation:

149.2 tokens/s

prompt processing (prefill):

3075.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.5-35B-A3B (MoE, 35B total, ~3.5B active) at 149.2 t/s decode and 3075 t/s prefill on an AMD Radeon AI PRO R9700 32GB. Setup is llama.cpp (Vulkan, RADV driver) with UD-Q4_K_XL quant, flash attention on, ubatch 2048, and a one-line rm_kq=1 patch; single GPU. The figures are the best RADV configuration (rm_kq=1 + ub2048); stock RADV gives 147.8 t/s decode and 2381 t/s prefill. AMDVLK with rm_kq=1 reaches 156.3 t/s decode but only 1823 t/s prefill. WikiText-2 perplexity is 6.9472 ± 0.046 across all configurations.

generation:

24.1 tokens/s

prompt processing (prefill):

154.7 tokens/s

quant:

UD-IQ4_NL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares ROCm and Vulkan performance for DeepSeek V4 Flash 0731 and Qwen 3.8 Flash Next on a Radeon AI PRO R9700 32GB plus Strix Halo in llama.cpp. DeepSeek V4 Flash 0731 with UD-IQ4_NL quant and a DSpark Q2K-Q4K drafter on ROCm reached 154.67 t/s prompt and 24.07 t/s generation on a 40,180-token needle prompt; Vulkan reached 131.97 t/s prompt and 20.44 t/s generation. Qwen 3.8 Flash Next with UD-Q4_K-XL and no speculative decoding on ROCm reached 566.10 t/s prompt and 17.96 t/s generation; Vulkan reached 477.33 t/s prompt and 18.99 t/s generation. User concludes ROCm still wins for DeepSeek while Vulkan has closed the gap, and for Qwen ROCm leads on prompt processing but Vulkan leads on token generation.

generation:

37.0 tokens/s

prompt processing (prefill):

1069.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 37 t/s generation and 1069 t/s prompt processing on two Radeon AI PRO R9700 32GB cards in a GMKtec EVO-X2 Strix Halo machine. Setup is llama.cpp with the ROCm build, RCCL enabled via -DGGML_HIP_RCCL=ON, and the UD-Q4_K_XL GGUF. The GPUs connect over NVMe-to-USB4 enclosures and run at PCIe 4.0 x4. Tensor parallelism was slower than a single card under the Vulkan build; the ROCm build restored a decode speedup, and RCCL restored full prompt processing. Single card measured 28 t/s generation and 1070 t/s prompt processing.

quant:

Q4_K_XL (GGUF)

User asks for real-world sustained t/s on an R9700 with Qwen3.8-27B Q4 at 64K context and above. User cites an AMD blog figure of 51.8 t/s and 5090 benchmarks showing degradation from 75 t/s at 4K to about 26 t/s at 64K. User also asks about MTP speculative decoding stability.

generation:

26.6 tokens/s

prompt processing (prefill):

279.3 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports tensor-read-lazy with mmap fitting a Q4 model on a system with an AMD Radeon AI PRO R9700, RX 9070 XT, and Radeon 8060S (Strix Halo). The model is 103.68 GiB with 176.94 B params. Prompt processing reaches 279.25 t/s and generation 26.63 t/s.

generation:

51.6 tokens/s

quant:

Q8_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 51.56 tokens per second over 5369 generated tokens with MTP on 2x R9700, writing a binary tree in javascript. Setup is llama.cpp built in the ROCm 10 docker image with unsloth's Q8_K_XL GGUF, tensor split across both cards, 255000 context, f16 KV cache and MTP drafting up to 3 tokens, using 27GB on each card. Draft acceptance is 0.72147. The user sees 37-50 tg/s generating text, spiking over 60tg/s when writing code.

generation:

35.0 tokens/s

prompt processing (prefill):

242.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports that 2 GPUs outperform 3 due to a slow lane. Setup is Qwen3.8 Flash Next with the Q4_K_M quant, variant AD-4.27bpw.

generation:

29.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Q4_K_XL on a Radeon AI PRO R9700 with llama.cpp at 29 TG/s, 76% of a theoretical 38 TG/s. The figure is a theoretical maximum TG/s calculation for dense models. An RTX 5090 with 1.8 TB/s bandwidth is also mentioned.

generation:

84.5 tokens/s

prompt processing (prefill):

2000.4 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 84.48 tokens per second generation and 2000.37 tokens per second prompt eval on a Radeon 9700 with 32GB VRAM. Setup is llama.cpp on Vulkan with the UD-Q5_K_M GGUF, -ngl 99, 262144 context, q8_0 KV cache, flash attention on, batch and ubatch size 2048, 8 threads and parallel 1. With MTP, which the user is now testing, generation is around 110 - 130 t/s and prefill drops to 1700 t/s. ROCm gave much faster prefill but slower generation.

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

on

User reports 40-67 t/s decode on 2x Radeon AI PRO R9700 32GB at 131,072 context. Setup uses an F16 KV cache. Prefill throughput is 410-1500 t/s. MTP draft acceptance ranges from 0.33-0.61.