- generation:
- 69.6 tokens/s
- prompt processing (prefill):
- 383.0 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 69.63 t/s generation on an RX 7900 XTX 24GB, with prompt processing at 383.02 t/s.
Setup is llama.cpp 2.61.0 (Vulkan/RADV) via LM Studio with Q4_K_M and q8_0 KV cache, 184320-token context, flash attention on, all layers on GPU and about 23.3 of 24 GiB VRAM in use.
Speculative decoding was on (MTP draft plus ngram, up to 2 draft tokens), so decode is inflated versus a plain run; draft acceptance was 0.558-0.68. A second 512-token run reached 64.09 t/s decode and 239.97 t/s prompt. One request at a time, nothing offloaded to system RAM.
- generation:
- 9.0 tokens/s
- prompt processing (prefill):
- 150.0 tokens/s
- quant:
- IQ2
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4.1 at 9 t/s decode average on an RX 7900 XTX 24GB plus W6800 32GB, with prefill around 150 t/s.
Setup is Dspark with mostly IQ2 weights and Q6 n-grams, about 350GB on disk, streamed from a PCIe gen3 NVMe drive on a Ryzen 9 3950X with 96GB dual-channel DDR4 at 3000MHz.
Decode varied widely: peaks of 19 t/s, usually around 13 t/s, dropping to 3 t/s when I/O was needed. User calls the model very smart but says it thinks a lot, so the slow average limits it to chatting.
- generation:
- 37.1 tokens/s
- prompt processing (prefill):
- 877.8 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 37.09 t/s generation and 877.84 t/s prompt processing on an AMD Radeon RX 7900 XTX under llama.cpp's Vulkan backend.
Setup is llama.cpp (build 10438) with a UD-Q4_K_XL GGUF of 16.68 GiB, run with GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 because Resizable BAR is disabled in the BIOS; without that variable token generation is slow.
At a 4096-token context depth the same run gives 36.33 t/s generation and 812.34 t/s prompt processing.
- generation:
- 78.5 tokens/s
- prompt processing (prefill):
- 523.3 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- q8_0
- flash attention:
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 78.53 t/s generation and 523.33 t/s prompt processing on a Radeon RX 7900 XTX 24GB.
Setup is llama.cpp 2.61.0 (Vulkan backend, LM Studio) with Q4_K_M GGUF and q8_0 KV cache at 106496 context, whole model on the single GPU (16.8GB of 24GB VRAM). Speculative decoding was on (MTP + ngram, spec-draft-n-max 2), so the generation figure is an effective speed with draft acceptance of 78/98.
Four runs, one request at a time: generation ranged 61.68 to 78.53 t/s across runs. Runs 3 and 4 repeated a cached prompt (588 of 592 tokens), so their prompt speeds are startup time, not prefill. The user runs this model as a daily driver for coding and agent work.
- generation:
- 41.7 tokens/s
- prompt processing (prefill):
- 1075.8 tokens/s
- quant:
- IQ4_XS (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6-27B at 41.70 t/s decode and 1075.8 t/s prompt processing on an RX 7900 XTX.
Setup is llama.cpp (ROCm/HIP backend, LTO and WMMA flash attention on) with IQ4_XS weights and F16 KV cache at 16K context, batch 1024 and ubatch 256.
The model is a 14.1 GiB MTP build; the user calls this the best overall configuration and notes LTO gives a 5-15% improvement.
- generation:
- 172.9 tokens/s
- prompt processing (prefill):
- 3290.9 tokens/s
- quant:
- Q4_0 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks a llama 7B Q4_0 model (6.74B params, 3.56 GiB) at 172.86 t/s generation and 3290.92 t/s prompt processing on a single AMD RX 7900 XTX.
Setup is llama.cpp (build a81a56957) with the Vulkan backend, flash attention on, ngl 100 and n_batch 512, on Arch Linux with RADV Mesa 25.3.1.
A dual-GPU run with 2x RX 7900 XTX was slower at 129.80 t/s generation and 2546.13 t/s prompt processing, which the user attributes to PCIe transfer overhead on small models.
- generation:
- 31.0 tokens/s
- quant:
- Q4_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 31 t/s on a single RX 7900 XTX 24GB with 32GB DDR5 6000 system RAM.
Setup is llama.cpp with Q4_K_M at 64K context.
User feels the setup is not optimized for coding on large projects and asks for resources explaining how quantization, KV cache types, context size, batch size, GPU offloading and expert hits affect memory use, speed and quality.
- prompt processing (prefill):
- 1300.0 tokens/s
- quant:
- Q3_K_S (GGUF)
- kv:
- Q8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen3.8 Flash-Next at around 50-60 t/s decode with 1300 t/s prefill on a dual RX 7900 XTX system.
Setup is Strata with a Q3_K_S GGUF and Q8 KV cache, running on the rest of the system while a Qwen3.8 27B Q4_K_XL model with 131k context and Q8 KV sits on the second card behind the chipset at gen3 x4.
The 27B gets around 40 t/s decode and 700-800 t/s prefill. An unused flag raises Flash-Next prefill to about 2300 t/s but makes inference unstable, and a layer split keeps decode the same while bringing prefill near 3000 t/s. User is experimenting with pi/hermes agent setups and is not satisfied with how the two models work together.
- generation:
- 90-120 tokens/s
- quant:
- IQ3_S (GGUF)
- kv:
- int8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports peak prompt processing of ~2500 t/s on prompts over 80k tokens and decode of 90-120 t/s with Swift 1.5 IQ3_S on two RX 7900 XTX cards in eGPU enclosures over OCuLink PCIe 4x4, on an HX370 mini PC with 64GB DDR5 5600.
Setup is Strata 1.4.4.1 with int8 KV cache, 262144 max context, MTP speculation, expert cache 7000, 20480 KV resident, 50 GiB resident budget, and MMAP to reduce system RAM pressure; vision is enabled.
The post is mainly a guide to building a tuned HIPblasLt table for the user's own hardware; the throughput figures are given as ranges.
- generation:
- 47.9 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8-27B at 47.9 tok/s on an RX 7900 XTX 24GB against Qwen3.8-Flash-Next at 36.8 tok/s on an MI50 32GB.
Setup is llama.cpp with UD-Q4_K_XL and F16 KV cache for the 27B; Strata with IQ3_S and fp16 KV cache for Flash-Next. Both ran through an OpenAI-compatible API with temp 1.0, top_p 0.95, top_k 20.
Speed figures cover AIME26 and LiveCodeBench generation (1.34M output tokens over 7.8 h for the 27B). The 27B scored 145/166 on the eval suite versus 151/166 for Flash-Next, a difference the user notes is not statistically significant (p = 0.18).
- generation:
- 16.0 tokens/s
- quant:
- IQ2_XXS (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, selecting experts on agentic coding sessions, then quantized with an imatrix built on the same sessions. The IQ2_XXS build (45.3 GB) runs on an RX 7900 XTX (24 GB) plus 32 GB DDR4-2133 with an i7-7700K over PCIe 3.0, at roughly 16 tok/s generation and roughly 950 tok/s prompt processing.
Setup is llama.cpp with IQ2_XXS GGUF weights and 384K context. The 2-bit version is experimental: it solved 39/48 attempts versus 48/48 for Q3_K and above, and when it fails it loops or keeps exploring until the turn limit. An anti-loop guard did not help (27/37 vs 30/37).
Evaluation used 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, 200 turns, in a sealed clean room with no network or git history. The code-scored expert mask needed x1.21 turns versus the full model, against x2.21 for a frequency mask and x2.29 for the public multi-domain REAP 50% mask. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.
- generation:
- 105-160 tokens/s
- quant:
- Q2
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next running locally on an AMD RX 7900 XTX at 500,000 context, with decode of 105–160 tok/s.
Setup is Strata with a Q2 quant; 66.4 GB of weights, 23.8 GiB resident in VRAM, 31.6 GB of expert weights and a 28.8 GB PLE table in system RAM.
Context held 506,849 tokens. Quality held between 81% and 84% with thinking off and 88–94% with it on across 4k, 64k and 256k. Decode fell 18% over that range. User calls it an incredible improvement over llama.cpp and Swift 1.5 27b dense, and now uses it as a daily driver.
- generation:
- 50.0 tokens/s
- prompt processing (prefill):
- 750.0 tokens/s
- quant:
- Q4
- kv:
- Q8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-context
User asks for hardware upgrade advice for running local LLMs, currently using an RX 7900 XTX with a Ryzen 5600X and 64GB DDR4. They report running Qwen3.8 27B at Q4 with 128k to 220k context and Q8 KV cache, getting up to 50 t/s generation and around 750 t/s prompt ingestion. They are considering 2-4 MI50 16GB cards or a second 7900 XTX, and ask about motherboard recommendations.
- generation:
- 83.0 tokens/s
- quant:
- 3.05bpw (EXL3)
- kv:
- Q8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-Flash-Next at 83 tok/s on code and 45-50 tok/s on prose on a single RX 7900 XTX 24GB with 60GB DDR5.
Setup is TabbyAPI with EXL3 3.05bpw, 128K context and 8-bit KV cache, 166 of 512 experts per layer on the GPU and 12 CPU threads, MTP drafting with 2 tokens.
A 28.7K-token code prompt prefills at 505 tok/s cold and decodes at 51-70 tok/s; without drafting the model runs 53 tok/s on code. The 4.05bpw variant reaches 68 tok/s on code. llama.cpp with IQ4_XS and --n-cpu-moe 36 is cited as a reference at 17.5 tok/s.
- generation:
- 157.0 tokens/s
- quant:
- EXL3 3.5bpw (EXL3)
- kv:
- Q8
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contextvision
User reports Qwen3.8-27B at 157 tok/s decode on an RX 7900 XTX 24GB with DFlash2 speculative decoding at 2K context.
Setup is a ROCm port of ExLlamaV3 with EXL3 3.5bpw weights, Q8 KV cache, and a DFlash2 5.0bpw draft model, served through TabbyAPI.
Decode drops to 147 tok/s at 32K and 107 tok/s at 99K; MTP gives 108 tok/s at 2K. Prefill is ~1180-1200 tok/s at 32K. Through the TabbyAPI OpenAI endpoint, code reaches ~186 tok/s and prose ~78 tok/s. The same GPU with llama.cpp and an IQ3_XXS GGUF decoded at 40-53 tok/s.
- prompt processing (prefill):
- 920.0 tokens/s
- quant:
- UD_Q3_XXL (GGUF)
- flash attention:
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports about 920 tokens/s prompt processing for Qwen3.8 Flash-Next MoE UD_Q3_XXL on two RX 7900 XTX cards, and about 1,600 tokens/s prompt processing for Qwen3.8 27B dense Q8_0 with tensor parallel.
Setup is a custom llama.cpp RDNA3 fork with ROCm 7.14, adaptive MTP draft depth up to 3, Q8_0 KV cache, layer split, and MoE experts offloaded to system RAM with a 288-slot GPU-resident expert cache.
The user notes the expert cache slows prompt processing to a crawl on their workload and they run with it disabled; with 288 experts they report a 95% hit rate and 45 t/s decode with MTP. The build is experimental and the figures are measured, not guaranteed.
- generation:
- 77.0 tokens/s
- prompt processing (prefill):
- 1816.0 tokens/s
- quant:
- Q8 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B Q8 at 77 t/s decode at 51K depth and 1,816 t/s prefill over a 51K prompt on four RX 7900 XTX cards after a series of ROCm and llama-server fixes.
Setup is 4x RX 7900 XTX on PCIe 4.0 x16 in a Lenovo P620, Q8 weights, a locally patched llama.cpp with MTP speculative decoding.
Before the fixes the same run gave 57 t/s decode at 51K and 1,454 t/s prefill; code decode went from 54-61 to 96-110 t/s. The patches are not yet published.