Local model performance on your hardware

generation:

46.4 tokens/s

quant:

IQ4_XS (GGUF)

kv:

F16

User reports Qwen3.8-27B at 46.4 t/s generation and 43.1 t/s prompt processing on 2× RX 7900 XT (20 GB each). Setup is llama.cpp with IQ4_XS, F16 KV cache, 8k context, MTP3 speculative decoding, Flash Attention, and ROCm 2.46.0. The 27B tests used roughly 700 input tokens and 1024 output tokens. Two separate IQ4 copies reached 103.5 t/s combined with 16 concurrent requests. Flash Next UD-IQ4_XS (93.7 GB) ran at 8.6 t/s on one GPU and 5.5–6.5 t/s on two GPUs. User asks whether the x1 PCIe slot, CPU/RAM offloading, or backend settings are the bottleneck.

generation:

52.0 tokens/s

prompt processing (prefill):

708.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

User benchmarks a local Qwen3.8-27B at 52.0 tok/s decode and 708 tok/s prefill on an RX 7900 XT, used as a coding executor for Claude across 51 runs. Setup is llama.cpp llama-server with IQ4_XS at 64K context, q8_0 KV cache, flash attention, full GPU offload, and MTP speculative decoding (draft n_max 2, p_min 0.75). The 27B was correct after review on 43/51 runs versus Sonnet 5's 45/51, tied on full-code and debug tasks, and failed all interface-only lifecycle and in-memory repository tasks. End-to-end throughput in the agent was 47.4 tok/s for the 27B and 107.8 tok/s for Sonnet, which cost $9.20 in API usage.