Local model performance on your hardware

generation:

15.4 tokens/s

prompt processing (prefill):

142.3 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

User reports DeepSeek-V4-Flash at 15.4 tok/s decode and 142.30 tok/s prefill at 2,048 tokens of context on a Ryzen AI Max+ PRO 395 (Strix Halo) with 128 GB unified memory. Setup is the ds4 engine with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF (81 GB), 32,768-token context, ROCm 7.1.1; decode falls 12% from 2K to 32K while prefill stays flat. A second model, Qwen3.8-Flash-Next UD-IQ4_XS (88 GB) under Strata 0.1.40, decodes at 39.0 to 40.9 tok/s and prefills a 6,786-token prompt at 343 to 367 tok/s, dropping to 30.4 tok/s each with two requests in parallel. The same engine measured 8.7 tok/s while another model's worker held 41 GiB of VRAM and 23 GB of RAM, against 39.0 tok/s on an exclusive box.

generation:

7.1 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash Q2 (~340 GiB MoE) running on a single AMD Strix Halo 128GB box with 124 GiB unified memory, reaching a highest completion-token rate of 7.12 tok/s including reasoning. Setup is the kyuz0 ds4 ROCm runtime with SSD expert streaming (--ssd-streaming-cache-experts), ROCm 7.2.4, 8k context, and a host-side patch. Small coding and Czech tests completed at request-wall medians of 29.2 s (Python), 26.4 s (bugfix), and 20.4 s (Czech); 64k prompts pass but 128k never produced a final answer in a 905 s bounded calibration. User stopped the experiment, calling it not a practical winner. A Three.js coding test produced 20,575 output tokens over ~83 minutes at ~3.9-4.2 end-to-end tok/s with zero accepted games. The same box measured Qwen3.8 Flash-Next at 44.66 tok/s short, 41.37 @ ~32k, 40.08 @ ~64k, and 38.03 @ ~126k with MTP decode, and GLM-5.3-Flash at 14.63 tok/s on ROCm versus 8.57 tok/s on Vulkan.