Local model performance on your hardware

generation:

18-22 tokens/s

quant:

UD-IQ1_M (GGUF)

flash attention:

on

User reports Qwen3.8-Flash-Next at roughly 18-22 tok/s generation on an M5 Pro MacBook Pro with 48GB unified memory. Setup is llama.cpp b10930 with Unsloth UD-IQ1_M GGUF, 4096 context, flash attention on, 12 experts on CPU, and --load-mode none --lazy-mode on to avoid Metal OOM. At 16384 context for an OpenCode test, a 10,634-token prompt took 503 seconds of prefill then generated 192 tokens in 16 seconds; a follow-up reused 10,825 cached tokens but 5,113 more tokens took 165 seconds, and a 28,081-token request was rejected. About 94% of inference time went to prefill, and the user calls it not practical for agentic coding on this setup.

generation:

19.0 tokens/s

prompt processing (prefill):

300-350 tokens/s

quant:

Q8_0 (GGUF)

User compares MLX and GGUF for Qwen3.8 27B on an M5 Pro, reporting generation at about 19 t/s. Setup is llama.cpp with Metal, which the user says now matches MLX prefill at roughly 300-350 t/s. The user questions whether MLX is still needed and mentions MTP and oMLX.