Local model performance on your hardware

generation:

98.8 tokens/s

prompt processing (prefill):

4245.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-8B Q4_K_M at 98.8 t/s decode and 4245 t/s prefill (pp512) on an RX 9070 XT with llama.cpp's HIP backend. Setup is llama.cpp b10290 official HIP build on one RX 9070 XT 16 GB, Q4_K_M weights, all layers on the GPU, Windows 11, no speculative decoding. The issue reports a Vulkan backend fault on the same card: Qwen3-8B Q4_K_M drops to 21.0 t/s decode and 791 t/s at pp128 there, and models with hidden size 4096 or more run 4.7-6.7x slower than on HIP, while Qwen3-4B is normal at 183.0 t/s. Qwen3-14B Q4_K_M runs 54.3 t/s on HIP against 8.1 on Vulkan.

generation:

7.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B Q4_K_M decoding at 7.89 t/s on an AMD RX 9070 XT under a pinned llama.cpp Vulkan build (b10964), while LM Studio's Vulkan llama.cpp build reaches 29.89-30.46 t/s on the same GPU and byte-identical GGUF. Setup is llama.cpp with Q4_K_M and q8_0 KV cache at 32768 context, flash attention on, single RX 9070 XT; llama-bench -p 128 -n 64 -r 1 gives 451.45 pp128 t/s and 7.89 tg64 t/s at -ngl 99. -ngl 0 (CPU) decodes at 8.00 t/s and -ngl 99 at 7.83-7.89 t/s, so GPU offload buys nothing; the iGPU alone gives 3.04 t/s. Through the managed runtime decode is 7.41-7.49 t/s with prompt eval 98-222 t/s. LM Studio loaded the model at 255744 context and was still about 4x faster. Root cause not established.

generation:

62.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B A3B at 62 t/s on an AMD RX 9070 XT 16GB, with vLLM on ROCm reaching 48 t/s in the same comparison. Setup is llama.cpp with the Vulkan backend, 32k context, on a Ryzen 7 9800X3D with 64GB DDR5; the MoE model spills part of its weights to system RAM. User notes the setup is early and numbers are not settled, that 16GB VRAM rules out big dense models, and that the local model needs supervision.

generation:

42.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 42 t/s on an AMD RX 9070 XT 16GB. Setup is llama.cpp with MTP2 speculative decoding and a 64K context, using about 1 GB of working RAM during evaluation. MTP2 delivered +41% to +53% throughput depending on the quantization, but the user notes the fastest configuration was not necessarily best for reasoning quality.

generation:

50.0 tokens/s

quant:

Q6 (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 50 t/s for Qwen 3.8 27B on an RX 9070 XT 16GB. Setup is llama.cpp (Vulkan build) with a Q6 quant, 131072 context, Q8 K cache and Q4 V cache, and DFlash2 speculative decoding with 7 max drafts. The rig also has an R9700 AI Pro 32GB and 64GB DDR5 6000; the user asks what speeds and flags others get, especially on Windows/AMD.

generation:

43.7 tokens/s

prompt processing (prefill):

700.8 tokens/s

quant:

IQ3_S (GGUF)

kv:

q5_0-q4_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B (Swift-1.5-GSQ-RCO IQ3_S with MTP) at 43.7 tok/s decode and 700.8 t/s prefill on an RX 9070 XT 16GB. Setup is llama.cpp (server-vulkan b11176) with IQ3_S weights and a q5_0-q4_1 KV cache at 65536 context, one card. Decode is the mean of 3 runs of a fixed 512-token generation; prefill is one cold 30,461-token prompt. At 131072 context the same Vulkan launch gives 37.4 tok/s decode and 701.6 t/s prefill, while the pinned ROCm image (b10884) cannot load at 128K and decodes 33.5 tok/s at 65536. The user attributes the ROCm failure to a missing FlashAttention vector kernel for the q5_0/q4_1 K/V pair, which promotes both to f16 and exhausts the 16GB card.

generation:

24.7 tokens/s

prompt processing (prefill):

2061.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

f8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 2061.11 t/s prompt and 24.70 t/s decode on two Radeon RX 9070 XT 16 GB cards under ROCm 7.2 on Windows. Setup is a llama.cpp fork with Q4_K_M GGUF, f8_e4m3 KV cache, FlashAttention, batch 8192 / ubatch 1024, layer split across both GPUs, one server slot, and cold prompt processing at 98,304 context (L3 lane). With MTP n3 the same L1 lane gives 1817.81 t/s prompt and 36.73 t/s decode at 46.8% acceptance. Vulkan rows are lower on prompt processing (L1 1569.17, L2 1470.80, L3 1243.99 t/s) and lead only the L1 decode rows. Linux ROCm 10 results reach 2253.06 t/s prompt with MXFP4-requant and 50.13 t/s decode with Q4_K_M plus MTP n3.

prompt processing (prefill):

706.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 706.0 t/s prompt processing on an AMD Radeon RX 9070 XT 16GB under ROCm. Setup is llama.cpp with Q4_K_M GGUF, flash attention enabled, -ncmoe 40 and -ub 512, on a Ryzen 9 5950X with 64 GB DDR4-3200. The same configuration under Vulkan reached 333.4 t/s prompt and 29.3 t/s generation; the user retired Vulkan in favour of ROCm.

generation:

62.0 tokens/s

quant:

Q6_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.5-9B at 62 t/s on an AMD Radeon RX 9070 XT (RDNA4/gfx1201) using llama-server with the Vulkan backend. Setup is llama.cpp with GGUF Q6_K_XL weights at 65536 tokens of context, single request, no speculation. The same model under vLLM ROCm 7.2 with FP8 weights reached 48 t/s, which the user attributes to vLLM lacking native gfx1201 kernel support and falling back to FP32 dequantization. The user reports llama-server Vulkan is 29% faster on this hardware and recommends it over vLLM until RDNA4 support lands upstream.

generation:

50.0 tokens/s

quant:

Q6 (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Swift at a ceiling of 50 t/s on an RX 9070 XT 16GB and a Radeon AI PRO R9700 32GB. Setup is llama.cpp with the Vulkan backend, a Q6 quant, 131K context, and Q8 KV cache for both K and V, with a DFlash2 drafter. User asks whether others get more than 50 t/s on AMD 9070 XT or R9700 hardware with a usable context size, and whether a different OS could add 30+ t/s.