Local model performance on your hardware

quant:

GSQ-RCO-IQ3_XXS (GGUF)

kv:

Q4_0

mtp (multi-token prediction):

on

User benchmarks Qwen3.8 27B on an RX 7600 XT 16GB, completing 15/15 tasks with 1.000 correctness in 348.5 seconds. Setup is llama.cpp HIP ROCm with the GSQ-RCO-IQ3_XXS GGUF and a Q4_0 KV cache at 114,688 context, fully in VRAM with no CPU offload. The same benchmark also ran Ornith-1.5-9B (15/15, 0.983, 184.9s) and K2-Horizon-7B (11/15, 0.909, 498.1s); the user calls Qwen3.8 27B the undisputed winner for agentic coding.

generation:

18.0 tokens/s

prompt processing (prefill):

141.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q8_0

mtp (multi-token prediction):

off

User reports Qwen3.8 27B at 18 t/s decode and 141 t/s prefill on a 1.8k-token prompt, running on an AMD Radeon RX 7600 XT 16 GB at 163,840 context. Setup is llama.cpp llama-server build 10480 with the unsloth Qwen3.8-27B-UD-IQ3_XXS GGUF (10.2 GiB), q8_0 KV cache, flash attention on, and Vulkan (RADV) backend. The model is a hybrid architecture where only 16 of 64 layers keep a full KV cache, so KV is about 5.3 GiB at this context. With MTP on the same model runs about 24 t/s at 98k context and 39 t/s at 124k. The user notes 160k is a VRAM-math ceiling, not a quality claim, and that decode drops to about 6 t/s if the GPU spills to system RAM.

generation:

20.0 tokens/s

prompt processing (prefill):

308.0 tokens/s

quant:

UD-Q2-K-XL

User reports Muse Glimmer 30B at about 308 t/s prompt processing and about 20 t/s generation on an RX 7600 XT 16GB. Setup is llama.cpp with ROCm, the UD-Q2-K-XL quant and DFlash speculative decoding. The run completed a coding task.