Local model performance on your hardware

generation:

12-15 tokens/s

quant:

IQ2_K (MLX)

kv:

KVarN K4/V4

User reports Qwen3.8-Flash-Next (125B) at 12-15 decode tokens per second on a 2021 M1 Max with 32 GB of unified memory. Setup is MoEspresso 3 with IQ2_K routed projections and KVarN K4/V4 KV cache, using Metal execution through the mlx-iqk library. Cache-Prior expert bias is enabled by default; prefill is unbiased. The user notes the speed holds only when no other memory-hungry applications are running. The user also reports an overall score of 84.3% on a reproducible 48-question benchmark, ahead of Claude Opus 4.8 xhigh (80.3%) and other hosted solutions.

generation:

15.8 tokens/s

prompt processing (prefill):

81.8 tokens/s

quant:

4bit

User compares MLX and llama.cpp on Qwen3.8-27B. MLX reaches 15.81 t/s generation and 81.76 t/s prompt processing. llama.cpp reaches 9.69 t/s generation and 99.61 t/s prompt processing. The llama.cpp run uses the UD-Q4_K_M quant.