Local model performance on your hardware

generation:

75.0 tokens/s

User reports Tiel-Coder-35B-A3B at around 75 tok/s decode on short prompts on an RX 7800 XT 16GB, dropping to about 49 tok/s at 100K context. Setup is shura.cpp, a llama.cpp fork with an automatic VRAM governor that keeps 1.5 GiB of VRAM free for the desktop, on a Ryzen 5 5600 with 32GB DDR4-3200 under Fedora and ROCm 7.2.4. Experts are cached on the GPU with the rest in system RAM. Prefill improved 16.3% at 28K context and 7.9% at 100K over the previous build, with short-context decode basically unchanged. KL divergence was 0.0346 against a 0.0343 self-comparison baseline. A 40-minute stress test of 384 requests mixing 28-34K prompts had no failed requests. Only AMD gfx1101 with ROCm was benchmarked; CUDA builds compile but are untested.

generation:

40-45 tokens/s

quant:

GSQ RCO

User asks whether any engines are optimized for consumer AMD hardware, noting most specialized engines target Nvidia. They report running Swift 1.5 GSQ RCO at about 40-45 TPS on an RX 7800 XT, with no engine named. The figure is given only as a range, and the user says they are happy with the setup and wonders how much more can be squeezed out of the hardware.

generation:

40.0 tokens/s

quant:

4bit

User reports Qwen3.8 27B at 4bit running at around 40 t/s on an RX 7800 XT 16GB with 32GB RAM, at a max context of 120k. User is considering upgrading to a used RX 7900 XTX for €900 to reach 250k context and replace AI subscriptions, but finds the current quality not great. User is not interested in Qwen3.8 Flash-Next.

generation:

35.0 tokens/s

quant:

UD-Q5_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 35 t/s on an RX 7800 XT 16GB, with a Ryzen 5 7600X and 32GB DDR5. Setup is llama.cpp with UD-Q5_K_M (about 25GB), which spills past the 16GB card. User asks whether to swap to a used 7900 XTX 24GB or add a used 7600 XT 16GB, and requests real t/s numbers for either setup under Vulkan or ROCm.

generation:

64.9 tokens/s

prompt processing (prefill):

985.8 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

int8

User reports Qwen3.8-Flash-Next at 64.9 tok/s aggregate decode and 985.8 tok/s prefill on an RX 7800 XT 16GB. Setup is Strata 0.1.42 with IQ3_XXS and int8 KV cache, 39.97 GiB of expert weights in host RAM and 5,048 experts (8.19 GiB) cached in VRAM. MTP was enabled with --spec 4. The 64.9 tok/s is aggregate decode over 54 requests; median per-request was 66.4 tok/s with a 56.1-82.8 tok/s range. Other fresh-prompt prefill readings were 912.9 tok/s at 6,731 tokens, 1,004.6 tok/s at 76,599 tokens, 791.0 tok/s at 116,392 tokens and 509.8 tok/s at 38,790 tokens. Across all 61 tail-skip requests aggregate decode was 52.7 tok/s. An informal comparison gave 50.8 tok/s without route tail skip and 56.7 tok/s with it.

generation:

110.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

turbo4

User reports Qwen3.6-35B-A3B at ~110 t/s on an AMD RX 7800 XT 16GB with MTP and turbo4 KV cache at 32K context. Setup is a NJannasch llama.cpp mtp-turboquant fork on ROCm 6.4.0 (gfx1101) with UD-IQ3_XXS GGUF, turbo4 KV cache, --spec-type draft-mtp, and --reasoning-budget 512. MTP acceptance was 86% at 32K context; turbo4 KV saved ~470 MB vs f16 at 32K. The user warns that a silent CPU fallback can drop generation to 5-10 t/s and that gfx1032 cards crash with this fork.

generation:

1.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.1 8B at 1.5 t/s on an AMD RX 7800 XT under Ollama, compared with 65 t/s in LM Studio on the same machine. Setup is Ollama 0.4.7 on Windows with the Q4_K_M quant, Adrenalin Edition 24.10.1 drivers. The user asks whether the gap is their setup or an Ollama limitation.

generation:

30.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0/q5_1

User reports Qwen 3.8 27B at ~30 t/s decode on an RX 7800 XT 16GB. Setup is llama.cpp with Vulkan, UD-IQ4_XS GGUF quant, q8_0/q5_1 KV cache, 100096 context, no MTP, single request, full GPU offload. User shares the setup because many believe Qwen 3.8 27B is infeasible on 16GB VRAM.