Local model performance on your hardware

generation:

57.4 tokens/s

prompt processing (prefill):

586.0 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 57.4 tok/s decode and 586 tok/s prefill on an M4 Max 128GB, where the model fits fully in memory. Setup is oMLX with the Jundot oQ4e+MTP MLX checkpoint; the 36GB paging configuration uses oMLX 0.7.0 expert offload with resident fraction 0.28 (143/512 experts per layer), PLE on SSD and MTP off. The same model on an M4 Max Studio 36GB measures 15.55 tok/s steady (n>=1024) and 13.0 tok/s short-run (n=128); exact replay on repeated prompts adds about 40% (14.3 to 20.1 tok/s). Other memory tiers are simulated: 48GB at 192 experts ~15.9 tok/s, 48GB at 128 experts ~11.8 tok/s, M4 Pro 32GB ~9.7 tok/s, M4 24GB ~4.9 tok/s.

generation:

53.3 tokens/s

prompt processing (prefill):

273.7 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 53.3 tok/s on a Mac Studio M4 Max 128GB. Setup is oMLX 0.6.3rc2 with oQ4e 4-bit affine weights, 262,144 context, ANE prefill and native MTP speculative decoding with k=3 draft tokens, single stream. Code generation measured 72.1 tok/s and prefill at 4K 273.7 tok/s. oMLX 0.6.1 with MTP gave 48.0 tok/s prose and 65.5 tok/s code; oMLX 0.6.3rc2 with ANE off gave 47.9 tok/s for both; vllm-metal 0.3.0 at 8-bit without MTP gave 13.2 tok/s code.

generation:

53.9 tokens/s

prompt processing (prefill):

679.1 tokens/s

quant:

oQ5e

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks three Qwen3.8-Flash-Next REAP builds for Apple Silicon on an M4 Max 40-core GPU with 128 GB unified memory at 4K context. Setup is oMLX with mixed-precision packages, 48 transformer layers, top-10 routing, a retained 512-expert native MTP predictor, and 262k configured context. The REAP-384 oQ5e build scored 97/100 HumanEval no-thinking and 61/100 LiveCodeBench no-thinking at 679.1 PP tok/s, 53.9 TG tok/s, and 72.8 GB peak memory. The REAP-384 oQ4e build reached 689.5 PP tok/s and 59.3 TG tok/s at 61.5 GB peak, while REAP-288 oQ4e reached 692.3 PP tok/s and 48.7 TG tok/s at 46.0 GB peak. Accuracy figures are 100-question stratified samples and throughput is workload-specific.

generation:

72.5 tokens/s

prompt processing (prefill):

1472.4 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B on a Mac with OMLX, comparing MTP enabled against disabled. MTP shows minimal speedup for the 35B MoE model but roughly 2x for the 27B dense model. Results include pp and tg t/s at various context lengths and batch sizes.