Local model performance on your hardware

reported speed:

25.0 tokens/s generation

quant:

GSQ-RCO

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 25 tok/s on an RX 6900 XT installed in a retired HP DL380p server with 172 GB DDR3 and 2x 10-core CPUs. Setup is Strata with llama.cpp and an HTTP router, GSQ-RCO quant at 64k context. The GPU is powered by an external desktop PSU; total draw is 250 W idle and about 450 W while inferring. User also reports Qwen3.8-27B with GSQ-RCO-IQ3_XXS and MTP at 192k context reaching 45 tok/s. The machine is a prototype; user plans a flexible riser and a proper GPU platform, and may add Nvidia P40s in the remaining slots someday.

reported speed:

30-45 tokens/s generation

quant:

IQ3_XXS (GGUF)

kv:

q8_0/q4_0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports running Qwen3.8-27b (Huihui abliterated, IQ3_XXS) on an RX 6900 XT 16GB with llama.cpp and ROCm 10. Prefill drops from 400 t/s to 260 t/s at 40k context; decode varies between 30 t/s and 45 t/s. Setup uses 100k context, KV cache q8_0/q4_0, flash attention, ngram-mod, MTP with n=2, and mmproj in system RAM; VRAM fills to 15.7/16 GB.