Local model performance on your hardware

generation:

4.6 tokens/s

prompt processing (prefill):

12.6 tokens/s

quant:

MQ4

User benchmarks a hipfire-native MQ4 build of KRAFTON's Raon-1-mini-preview (28.8B/4.3B MoE) on a 20 GB RX 7900 XT with ROCm 7.1, getting 12.6 t/s prefill at pp512 and 4.6 t/s decode. Setup is hipfire with an MQ4 quant; the numbers are single runs. The same model under llama.cpp with the official Q4_K_M GGUF reached 2,055 t/s pp512, 1,477 t/s pp8192 and 47.4 t/s decode, roughly 200x faster prefill and 10x faster decode. The user attributes the gap to hipfire handling recurrent state token by token with no batched prefill kernel. The MQ4 file is 1.29 GB smaller than the official GGUF and agrees with it at mean KL 2.06 and 51.6% top-1; 65K context is the ceiling on 20 GB, and stock hipfire v0.4.2.1 needs a patch to serve this model.

generation:

36.5 tokens/s

quant:

MQ4

User reports the Swift-1.5 fine-tune of Qwen3.8-27B at 36.5 tok/s decode on an RX 7900 XT 20GB, with MTP speculative decoding (acceptance τ≈1.94). Setup is the hipfire runtime (v0.4.2.1+) with MQ4 weights and ROCm 7.1; the model fits in ~17.5 GB VRAM at 8K context, and 49K context with MTP is the recommended daily setting. Prefill holds ~1,100 tok/s at 16K, and decode is 52–59 tok/s at 16–49K context. Offloading layers to system RAM past that drops decode to ~16–31 tok/s. The user claims ~2.2x faster end-to-end than Q4_K_S GGUF on llama.cpp, and notes the .mq4 files only load in hipfire and were tested only on gfx1100.

generation:

54.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next at 54 t/s decode and 1,100 t/s prefill on a single AMD RX 7900 XT with 20GB VRAM plus 96GB system RAM. Setup is Strata with Huihui UD-IQ4_XS quant (2/3 experts IQ3_S, rest Q8+) and KV streaming for the full 262k context window, using auto:32768 context on ROCm 10.1. User is happy with the result but asks for advice on speeding up prefill further.

generation:

7.0 tokens/s

kv:

Q8

User reports about 7 t/s running Qwen3.8 27B Uncensored on an RX 7900 XT 20GB with 32GB DDR5 and a Ryzen 5 7600X. Setup used Hermes with LM Studio as a server, Q8 KV cache to fit the model in 20GB VRAM; the initial Hermes default configuration offloaded most work to CPU. User says the experience was poor, with overthinking loops and slow responses, and asks whether the hardware is the problem or their configuration.

generation:

15.0 tokens/s

prompt processing (prefill):

415.0 tokens/s

quant:

Maya-S

User reports GLM-5.3-Flash at ~415 tok/s prefill and ~15 tok/s decode on an RX 7900 XT 20 GB. Setup is Project Maya (v1.0.11, experimental AMD ROCm port) with Maya-S quant, 8K context, ROCm 7.2, 4K-token prompts, greedy sampling; the dGPU box has 192 GB of RAM, with hot experts in VRAM, next in pinned RAM and the rest on NVMe. Other configurations: Radeon AI PRO R9700 32 GB at ~500 tok/s prefill and ~20 tok/s decode; Strix Halo (Ryzen AI Max+ 395, 8060S, 128 GB) at ~210 tok/s prefill and ~18 tok/s decode; R9700 + 7900 XT with layer split and MTP drafting at ~490 tok/s prefill and ~34 tok/s decode. With less RAM more experts come from the SSD and decode drops.

generation:

46.4 tokens/s

quant:

IQ4_XS (GGUF)

kv:

F16

User reports Qwen3.8-27B at 46.4 t/s generation and 43.1 t/s prompt processing on 2× RX 7900 XT (20 GB each). Setup is llama.cpp with IQ4_XS, F16 KV cache, 8k context, MTP3 speculative decoding, Flash Attention, and ROCm 2.46.0. The 27B tests used roughly 700 input tokens and 1024 output tokens. Two separate IQ4 copies reached 103.5 t/s combined with 16 concurrent requests. Flash Next UD-IQ4_XS (93.7 GB) ran at 8.6 t/s on one GPU and 5.5–6.5 t/s on two GPUs. User asks whether the x1 PCIe slot, CPU/RAM offloading, or backend settings are the bottleneck.

generation:

115.0 tokens/s

quant:

GPTQ-Int4 (GPTQ)

kv:

fp16

User reports Qwen2.5-7B-Instruct-GPTQ-Int4 at 115 tok/s single-stream decode on a Radeon RX 7900 XT (gfx1100) under native Windows with AMD ROCm. Setup is vLLM v0.19.1 with GPTQ Int4 weights, fp16 KV cache, native exllama W4A16 GEMM and hipGraph decode capture (FULL_DECODE_ONLY), gpu_memory_utilization=0.9, verified spill-free. Aggregate throughput scales with concurrency: ~73 tok/s at batch 4, ~232 at batch 16, ~358 at batch 32. KVarN KV-cache quantization (K 4-bit / V 2-bit) gives 4.74x KV capacity (999,296 vs 210,784 tokens) at 74.7 tok/s, about 35% slower. A community-validated RX 7800 XT (gfx1101) reached 81.9 tok/s on the same model.

generation:

43.0 tokens/s

quant:

w4a16 (W4A16)

User reports gemma-4-31B at 43 tok/s decode on 2× RX 7900 XT with tensor parallelism. Setup is vLLM with w4a16 weights, TP=2, both cards drawing 265 W at the same time. The machine is a VFIO virtual machine with no P2P and cross-die PCIe 3.0, measured with no PCIe atomics. A 26B MoE reaches 108 tok/s at short context on the same pair.

generation:

52.0 tokens/s

prompt processing (prefill):

708.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

User benchmarks a local Qwen3.8-27B at 52.0 tok/s decode and 708 tok/s prefill on an RX 7900 XT, used as a coding executor for Claude across 51 runs. Setup is llama.cpp llama-server with IQ4_XS at 64K context, q8_0 KV cache, flash attention, full GPU offload, and MTP speculative decoding (draft n_max 2, p_min 0.75). The 27B was correct after review on 43/51 runs versus Sonnet 5's 45/51, tied on full-code and debug tasks, and failed all interface-only lifecycle and in-memory repository tasks. End-to-end throughput in the agent was 47.4 tok/s for the 27B and 107.8 tok/s for Sonnet, which cost $9.20 in API usage.

generation:

15.0 tokens/s

User reports Qwen3.8 Flash Next at 15 t/s on an RX 7900 XT with 20 GB VRAM and 96 GB DDR5 RAM. User asks whether to add another 96 GB of system RAM or a second RX 7900 XT for running local LLMs such as Qwen3.8 27B or Qwen3.8 Flash Next.