Local model performance on your hardware

prompt processing (prefill):

920.0 tokens/s

quant:

UD_Q3_XXL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports about 920 tokens/s prompt processing for Qwen3.8 Flash-Next MoE UD_Q3_XXL on two RX 7900 XTX cards, and about 1,600 tokens/s prompt processing for Qwen3.8 27B dense Q8_0 with tensor parallel. Setup is a custom llama.cpp RDNA3 fork with ROCm 7.14, adaptive MTP draft depth up to 3, Q8_0 KV cache, layer split, and MoE experts offloaded to system RAM with a 288-slot GPU-resident expert cache. The user notes the expert cache slows prompt processing to a crawl on their workload and they run with it disabled; with 288 experts they report a 95% hit rate and 45 t/s decode with MTP. The build is experimental and the figures are measured, not guaranteed.

generation:

77.0 tokens/s

prompt processing (prefill):

1816.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Q8 at 77 t/s decode at 51K depth and 1,816 t/s prefill over a 51K prompt on four RX 7900 XTX cards after a series of ROCm and llama-server fixes. Setup is 4x RX 7900 XTX on PCIe 4.0 x16 in a Lenovo P620, Q8 weights, a locally patched llama.cpp with MTP speculative decoding. Before the fixes the same run gave 57 t/s decode at 51K and 1,454 t/s prefill; code decode went from 54-61 to 96-110 t/s. The patches are not yet published.

generation:

44.5 tokens/s

prompt processing (prefill):

404.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0-q5_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 44.5 tok/s decode and 404 tok/s prefill on an RX 7900 XTX 24GB. Setup is llama.cpp (ROCm host build b11160 for gfx1100) with IQ3_S GGUF weights and q8_0/q5_1 KV cache, running the full 262,144-token window with MTP speculation on. Context gate recalled at 213,570 prompt tokens in 528.7 s. Paired MTP measurements for this launch are spec-off 11.2 tok/s versus 23.3 tok/s at 240K fill. The desktop shares about 1.2 GiB of the card, so a headless box may decode slightly faster.

generation:

68.9 tokens/s

prompt processing (prefill):

380.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0/q5_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 68.9 t/s on an RX 7900 XTX 24 GB at empty context with MTP n=3 speculative decoding, and 37.2 t/s without MTP. Setup is llama.cpp (ROCm, hip-kvmix) with IQ3_S GGUF and q8_0/q5_1 KV cache at 262,144-token context. At 240K fill with MTP the user reports 18.6-26.9 t/s (mean 23.3) and 380 t/s prefill, with 22,630 MiB peak VRAM.

generation:

23.8 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 23.8 tok/s decode on a single Radeon RX 7900 XTX 24 GB. Setup is a llama.cpp fork with a 131,072-token context and single stream (parallel 1), no speculation. With draft-mtp plus ngram-map-k speculation the same setup reaches 47.6 tok/s. The card runs a 272 W power cap (factory 303 W), a -75 mV undervolt and max SCLK 2200 MHz.

generation:

23.7 tokens/s

prompt processing (prefill):

429.7 tokens/s

quant:

IU4 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash at 429.68 t/s prefill and 23.74 t/s decode on an RX 7900 XTX 24GB plus Strix Halo 128GB over OCuLink. Setup is a custom llama.cpp runtime with IU4 GGUF weights, F16 KV cache, MTP depth 3, batch/microbatch 1536, eight CPU threads and one inference slot, targeting 262,144 tokens of context. The full-context request processed 261,888 prompt tokens in 609.49 seconds and generated 128 tokens in 5.35 seconds. A HermesAgent-20 agent workload with prompt caching on reports 590.24 t/s prefill and 50.00 t/s decode across 120 requests, with a decode range of 42.07-56.50 t/s for responses of at least 100 tokens. The user notes the approximately 50 t/s decode is an agent-workload result, not a filled-256K measurement.

generation:

126.1 tokens/s

prompt processing (prefill):

3785.1 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks llama 7B Q4_0 at 126.09 t/s generation and 3785.07 t/s prompt processing on a single AMD Radeon RX 7900 XTX. Setup is llama.cpp with the ROCm backend, Q4_0 quant, all layers offloaded (-ngl 99), and Flash Attention enabled. A second RX 7900 XTX gives 138.57 t/s generation and 3800.57 t/s prompt processing on the same model. Dual-GPU 7B is slower (94.13 t/s generation) as the model is too small to benefit. Llama 3.1 70B Q4_K_M across both GPUs reaches 13.37 t/s generation and 341.41 t/s prompt processing with Flash Attention.

generation:

142.8 tokens/s

prompt processing (prefill):

4331.2 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 142.76 t/s generation and 4331.21 t/s prompt processing on an RX 7900 XTX after a llama.cpp Vulkan int8 coopmat1 matmul patch. Setup is llama.cpp with the Q4_0 GGUF, Vulkan backend, flash attention enabled, n_ubatch 1024, and all layers offloaded. Before the patch the same run measured 135.64 t/s generation and 3410.53 t/s prompt processing. At 8192 context the figures are 125.29 t/s generation and 2885.67 t/s prompt processing; at 16384 context they are 120.82 t/s and 2402.13 t/s.

generation:

30.1 tokens/s

prompt processing (prefill):

422.3 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Gemma 4 31B on an RX 7900 XTX, reporting 30.13 t/s generation and 422.27 t/s prompt processing with ROCm. Setup is llama.cpp on Windows 10 with the UD-Q4_K_XL GGUF, flash attention, 32K context, all layers on GPU, and a Q4_0 KV cache. Vulkan single-GPU gave 24.81 t/s generation and 117.29 t/s prompt, while Vulkan multi-GPU with an added RTX 3080 gave 26.18 t/s generation and 74.69 t/s prompt. User notes ROCm started at maximum generation speed and declined with context, whereas Vulkan briefly sped up before declining.

generation:

24.0 tokens/s

prompt processing (prefill):

920.0 tokens/s

quant:

Q3_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Next Q3_K_XL at 24/27 tk/s on prose and 40+ tk/s on code with MTP on two 7900 XTX cards, with prefill at 920 tk/s pp8192. Setup is a custom llama.cpp build optimized for the 7900 XTX, with RAM offload and without the MoE expert cache. The user also benchmarks Qwen3.8 27B Q8_0 with tensor parallel at 1600 tk/s pp8192, 60/65 tk/s prose and 100+ tk/s code, and Qwen3.6 27B Q4_K_M on a single card with MTP at 1020 tk/s pp8192, about 58/60 tk/s prose and 75/80 tk/s code.

generation:

12.0 tokens/s

prompt processing (prefill):

141.4 tokens/s

quant:

UD-IQ3_XXS

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports prompt processing in the high 140s t/s, dropping to the mid 80s at 60k context, with generation at ~11.9 t/s. Setup uses the ROCm 7.2.4 backend on 1x 7900 XTX 24GB plus 3x MI60 32GB. One MI60 failed.

generation:

72.1 tokens/s

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-27B against Qwen3.6-27B on a fact-extraction task, where Qwen3.8 scores 0.7030 F1 against Qwen3.6's 0.7177, a statistical tie. Decode throughput falls from 85.6 to 72.1 t/s, about 16%, under the closest saved configs, though the runs used different llama.cpp builds. Qwen3.8 produces shorter answers, lowering end-to-end latency. The user expected larger gains from public benchmarks, and sees small gains on task-specific tests but massive gains only on benchmarks the model was trained on.

generation:

25.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Q4_K_M at 25-35 t/s on an RX 7900 XTX at 80k+ context, dropping to 20-21 t/s at 150k-180k context. Setup is llama.cpp with MTP enabled and a q8_0 KV cache with a q5_1 K cache. User expresses frustration over the speed and power consumption.

generation:

27.0 tokens/s

prompt processing (prefill):

573.0 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports llama.cpp benchmarks on a dual-GPU setup of an RTX 4090 laptop and an RX 7900 XTX over TB4, reaching full 262k context with +70% generation and +40% prefill. Setup uses the CUDA backend for the NVIDIA card and Vulkan for the AMD card, with a q8_0 KV cache to enable 262k context at some cost in speed. MTP speculative decoding gave +50-70% generation but halved prefill, and the user filed a bug. A Q6_K_XL quant over Q8_0 gave +9% generation and more context on CUDA, and rebuilding to a newer llama.cpp gave +13% generation for free.

generation:

14.4 tokens/s

prompt processing (prefill):

448.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a custom llama.cpp Vulkan backend patch for Qwen3.8-Flash-Next improves prompt processing from 346 to 448 t/s and generation from 13.3 to 14.4 t/s at 2.7k context. At 131k context, prompt processing improves from 195 to 263 t/s and generation from 9.1 to 10.4 t/s. The patch fixes a top-k fallback to CPU in sparse attention. Greedy outputs are identical, the needle is found at 131k, and images are read correctly.

generation:

11.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 11 t/s on Qwen 3.8 with 2x RX 7900 XTX and 128 GB DDR4. User compares the result to 3090s and 9700s.