Local model performance on your hardware

generation:

21-25 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports Ornith-1.5-122B decoding at ~22 tok/s (21-25 across runs) on real coding output on a single RX 7900 XTX 24 GB, with 32 GB DDR4 and an i7-7700K. Setup is a patched Strata on ROCm with unsloth's UD-IQ2_XXS (42 GB), experts split between VRAM, an 18 GB RAM tier and the SSD, so weights stream from system RAM and disk. Prompt processing on an 8.9K-token prompt ran at 407 tok/s (22 s), and decode at 9K context fell to 12.2 tok/s. The model is Qwen3.5-MoE based with KLD vs BF16 of 0.1225.

prompt processing (prefill):

1300.0 tokens/s

quant:

Q3_K_S (GGUF)

kv:

Q8

User reports Qwen3.8 Flash-Next at around 50-60 t/s decode with 1300 t/s prefill on a dual RX 7900 XTX system. Setup is Strata with a Q3_K_S GGUF and Q8 KV cache, running on the rest of the system while a Qwen3.8 27B Q4_K_XL model with 131k context and Q8 KV sits on the second card behind the chipset at gen3 x4. The 27B gets around 40 t/s decode and 700-800 t/s prefill. An unused flag raises Flash-Next prefill to about 2300 t/s but makes inference unstable, and a layer split keeps decode the same while bringing prefill near 3000 t/s. User is experimenting with pi/hermes agent setups and is not satisfied with how the two models work together.

generation:

90-120 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

User reports peak prompt processing of ~2500 t/s on prompts over 80k tokens and decode of 90-120 t/s with Swift 1.5 IQ3_S on two RX 7900 XTX cards in eGPU enclosures over OCuLink PCIe 4x4, on an HX370 mini PC with 64GB DDR5 5600. Setup is Strata 1.4.4.1 with int8 KV cache, 262144 max context, MTP speculation, expert cache 7000, 20480 KV resident, 50 GiB resident budget, and MMAP to reduce system RAM pressure; vision is enabled. The post is mainly a guide to building a tuned HIPblasLt table for the user's own hardware; the throughput figures are given as ranges.

generation:

105-160 tokens/s

quant:

Q2

User reports Qwen3.8-Flash-Next running locally on an AMD RX 7900 XTX at 500,000 context, with decode of 105–160 tok/s. Setup is Strata with a Q2 quant; 66.4 GB of weights, 23.8 GiB resident in VRAM, 31.6 GB of expert weights and a 28.8 GB PLE table in system RAM. Context held 506,849 tokens. Quality held between 81% and 84% with thinking off and 88–94% with it on across 4k, 64k and 256k. Decode fell 18% over that range. User calls it an incredible improvement over llama.cpp and Swift 1.5 27b dense, and now uses it as a daily driver.

generation:

45-50 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8 Next at 45-50 t/s on an RX 7900 XTX with slow DDR4 system memory, using Strata with IQ3_XXS weights just under 80GB. Setup is Strata with IQ3_XXS weights, MTP speculation enabled, and CPU offload to system RAM. Llama.cpp with tuning maxed out around 22.5 t/s on the same rig. User notes similar results for 12GB and 16GB cards, faster numbers for DDR5 owners, and recommends against the Q2 weights.