Oct 8, 2026
Summary
User reports Qwen3.8-Flash-Next 4-bit at ~25 tok/s decode on an Apple M3 Ultra with 512 GB unified memory.
Setup is oMLX 0.6.3 serving the mlx-community 4-bit weights, the whole model in unified memory, non-streaming chat completions on localhost with max_tokens 400-512 and default sampling.
Speed was taken as completion tokens over wall time: two short-prompt runs at low reasoning effort gave 24.6 and 25.6 tok/s, one at xhigh effort 24.2 tok/s, and a ~1.5k-token prompt 17.9 tok/s including prefill.