Local model performance on your hardware

generation:

18.5 tokens/s

prompt processing (prefill):

381.6 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8 27B at 18.5 t/s generation and 381.6 t/s prompt processing on an M5 Pro with 64GB unified RAM. Setup is llama.cpp with the Unsloth UD-IQ4_XS quant, run via llama-bench at -p 512 -n 128. The source gives 18.45 +/- 0.03 tok/s and 381.62 +/- 0.18 tok/s; the generation figure is not a post-512-prompt end-to-end run, and context limit, KV precision and offload settings are unstated.

generation:

27.6 tokens/s

prompt processing (prefill):

367.0 tokens/s

quant:

Q4_0 (GGUF)

User reports Qwen3.8 Flash-Next at ~27.6 t/s generation and ~367 t/s prompt processing on an M5 Pro 64GB, with the 95.5 GiB model streamed from SSD. Setup is a llama.cpp fork with expert streaming, a draft head, depth 3 with a 0.3 confidence floor, sparse attention, and Metal MoE fusion, using a bartowski Q4_0 checkpoint with Q8_0 output layer and unsloth UD-IQ4_XS spliced in. Generation drops to 18-18.6 t/s with the draft head off and ~20.6 t/s at 29k context in real chat use. Paired 40-chunk perplexity improved from 5.2777 to 4.3148.