Local model performance on your hardware

generation:

11.5 tokens/s

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 177B running at 11-15 tok/s on a single RTX 5070 12GB with 32GB DDR4-2400 RAM and a Ryzen 5 5600GT on Windows. Setup is llama.cpp with UD-IQ3_XXS quant and expert streaming from system RAM, using one file handle per worker and a page-locked hot-expert tier. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. Normal conversations run 14-15 tok/s, and a long coding prompt generated 4,892 tokens at 10.15 tok/s. A video shows around 16 minutes for 10k tokens at 10.41 tok/s.

prompt processing (prefill):

1800.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports that after AUR llama.cpp-cuda package deletion, binary packages (ggml-cuda and llama.cpp) are ~10% speed, with model loaded to GPU but processing on CPU. Custom build achieves 1800 t/s prefill vs 150 t/s. GPU is RTX 5070.

generation:

15.0 tokens/s

prompt processing (prefill):

104.7 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 15 t/s on an RTX 5070 12GB, with prompt processing at 100-120 t/s. Setup is llama.cpp with a 3 bpw IQ3_XXS GGUF (76GB total, 47GB sharded into VRAM+RAM, rest on SSD) at up to 128K context. Generation held 13.8 t/s at 1K, 14.6 t/s at 8K, 14.0 t/s at 20K, and 11.3 t/s at 90-100K context. User rates output quality as matching Unsloth Q6-Q8.

generation:

22.0 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 22.0 t/s generation and 24.96 t/s prompt processing on a single RTX 5070 with 12 GB of VRAM. Setup is llama.cpp with the IQ1_S quant at 10,000 tokens of context.