Local model performance on your hardware

generation:

43.0 tokens/s

User reports Qwen3.8 27B (abliterated) at ~43 tok/s on an M4 Max 128GB. Setup is mlx-serve 26.10.2 with MTP enabled; the 27B build is 16GB. The user also lists other abliterated models in their picker: Smart 35B at ~106 tok/s (24GB), Vision 27B at ~22 tok/s (15GB), Coder 80B at ~78 tok/s (49GB), DeepSeek-V4-Flash 284B at ~26 tok/s (75GB), and Flash-Next 125B at ~90 tok/s (77GB). They ask for suggestions on better packs and MTP or mlx-serve settings for this hardware.

generation:

60.0 tokens/s

prompt processing (prefill):

730.0 tokens/s

quant:

mixed 4-8bit (MLX)

User reports Qwen3.8-Flash-Next at ~60 tok/s serial decode and ~730 tok/s prefill on an M4 Max 128GB. Setup is mlx-serve 26.8.11 with a mixed 4-bit/8-bit MLX pack, ~75 GB resident, prefix cache on, images included. With MTP enabled decode reaches 78 tok/s (+41% on code, a few percent slower on prose). A needle at 24.8k tokens was recovered with sparse attention engaged. The pack includes the MTP head and vision tower; the 51B n-gram table ships as a 32 GB mmapped file that is not resident. The author recommends a re-quantized iQ-MLX-4.7bpw version instead.

generation:

73.8 tokens/s

quant:

Q4 (MLX)

User reports Qwen3.8 Flash-Next at 73.8 tok/s on an M4 Max with 128 GB unified memory. Setup is mlx-serve with Q4 affine routed experts, BF16 core and verification head, native MTP at depth 4, and the full 262,144-token context; the 51.2 GB n-gram lookup table streams from SSD. Matched warm workloads measured 71-74 tok/s for code and technical prose, with serial decode at 33.2 tok/s and a 2.22x MTP speedup. Long-context prefill reached 280.7 prompt tok/s at 248,445 tokens.