Local model performance on your hardware

generation:

53.0 tokens/s

prompt processing (prefill):

1620.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at 53 tokens/s generation and 1,620 tokens/s prompt processing on an RTX 5070 12GB with 64GB RAM. Setup is the Strata engine with IQ3_S quantization at 128K context; the model's experts are split across GPU, RAM and SSD. The page also lists Q2_0 at 94/2,650, IQ2_XS at 79/2,090, IQ3_XXS at 62/1,750 and Coder at 55/2,180 tokens/s on the same card, and RX 9070 XT 16GB figures of 60/1,160 (Q2_0), 52/1,110 (IQ2_XS) and 44/1,420 (Coder). Unsloth UD-Q4_K_XL is given as 7-8.5 tokens/s on a 64GB PC.

generation:

44.8 tokens/s

prompt processing (prefill):

414.0 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8-Flash-Next at 44.8 t/s generation and 414 t/s prompt processing on a 12GB RTX 5070 at 128K context. Setup is a custom Strata inference engine with the IQ3_XXS GGUF quant, 64GB DDR5-5600 and a Ryzen 5 7600 on Windows. The engine is CUDA-only and requires 47GB minimum in RAM+VRAM for this quant. A Q2_0 quant reached 65.1 t/s generation and 543 t/s prompt processing, and IQ2_XS reached 52.0 t/s generation and 472 t/s prompt processing. The user previously measured 15 t/s output and 100-120 t/s prompt processing with llama.cpp on the same hardware.