Local model performance on your hardware

generation:

81.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0/q4_0

User reports Qwen3.8-27B ASCII-Condensed UD-IQ4_XS at about 81 t/s code generation on an RX 9070 16 GB card with 32 GB system RAM, running on Windows. Setup is achyudh's llama.cpp ROCm KV-streaming fork with q8_0 K / q4_0 V cache, 262144 context, --kv-stream with 300/1800 MiB resident KV, MTP n=4 plus n-gram speculation, and 6 threads. Decode was 75 t/s at 9k and 95 t/s at 21k tokens, prefill about 1000+ t/s; these are single runs on short prompts, not filled 64k-262k contexts, and output quality was not checked. The user notes their numbers are below the fork README's ~100+ t/s at 16k-64k.

generation:

35-50 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

q4_0

User reports Qwen3.8-27B at 35-50 tok/s on an RX 9070 16GB, used for coding help. Setup is Unsloth Studio with UD-IQ3_XXS GGUF and q4_0 KV cache, micro-batch 256, context set to 128000, taking 15.50 GiB of VRAM. The user says the model is stuck at very high thinking and asks about offloading to 32GB DDR5 or SSD, noting 27B spilled outside the GPU is horribly slow and 30B-A3B Q3/IQ3 variants run fast but seem stupid.

prompt processing (prefill):

600.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

q4_0

User reports Qwen3.8 27B at 600 t/s prefill and 22-26 t/s generation on an RX 9070 16GB. Setup is llama.cpp with IQ3_S GGUF and q4_0 KV cache at 131072 context, using about 15.2GB of VRAM. MTP is not active because it did not improve performance and used more VRAM; generation starts at 26 t/s and degrades to 22 t/s at full context.

generation:

140.0 tokens/s

quant:

4-bit

User reports a 21M model with a 16.8M-row lookup table (6.4B table parameters, 33M used per token) writing ~140 tok/s on an RX 9070, with the 4-bit table memory-mapped from an NVMe SSD and using 0.4 GB of VRAM. The model matches a 114M dense model trained on the same 500M Wikipedia tokens. Reading long prompts from the SSD is slow because every missed row costs a 4 KB page. Triton kernels run unchanged on the Radeon, an MI350X and H100/H200. Bolting a table onto Qwen3.5-0.8B did not help. Caveats: tiny model, one seed for the big runs, fluent but factually made-up text.

generation:

31.7 tokens/s

quant:

GSQ-RCO (GGUF)

User reports Qwen3.8 27B GSQ-RCO at 31.7 t/s on a single AMD Radeon RX 9070 with 16 GB VRAM at 96k context, of which 38k were used. The GSQ-RCO GGUF quant scored 88.84/100 on llm-bench.io, the highest quality score recorded in over 1100 community benchmarks. The user notes it is not the fastest but sufficient for most coding tasks and praises the quant for preserving quality.

generation:

60.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

K 8_0 / V 4_0

User reports Qwen3.8 27B at about 60 t/s on a 16 GB RX 9070 XT at 128k context. Setup uses the ISTA-DASLab GSQ-RCO IQ3_XXS GGUF with a HermiHg DFlash2 Q2_K_S draft model for speculative decoding, and a K 8_0 / V 4_0 KV cache. User says this beats the built-in MTP, which multiplies VRAM use, and calls it one of the best single-thread choices on a 16 GB GPU. Speculative decoding is more sensitive to GTT overflow, so disabling it is faster when exceeding VRAM on a DDR5 PCIe 5 system.

generation:

73.0 tokens/s

quant:

Q1_0

kv:

kvarn4

User reports 43.4 t/s with a q8_0 KV cache and 73.0 t/s with a kvarn4 KV cache at 120K context. The q8_0 cache used about 13.1 GB total and the kvarn4 cache about 9.8 GB total. The kvarn4 cache is 68% faster and uses 3.3 GB less VRAM.

generation:

40.7 tokens/s

prompt processing (prefill):

778.9 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

Q8

User reports Qwen3.8-27B-UD-IQ3_XXS.gguf at 40.68 t/s generation and 778.91 t/s prompt processing at 131,072 context, against 13.36 t/s and 137.87 t/s at 65,536 context. Setup is llama-server with flash-attn, a q8_0 KV cache, Vulkan0, and speculative decoding with draft-mtp and ngram-mod. User asks why the smaller context degrades performance.

generation:

46.9 tokens/s

prompt processing (prefill):

398.4 tokens/s

quant:

UD-Q5_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

on

User reports generation around 45-52 t/s on two RX 9070 XTs with ROCm. Setup is MTP with spec-type = draft-mtp and spec-draft-n-max = 2, using the unsloth UD-Q5_K_XL GGUF. Prompt t/s varies, and draft acceptance rate is around 0.8-0.99. User praises the speed, smarts and steerability for agentic coding tasks.