Local model performance on your hardware

generation:

59.6 tokens/s

prompt processing (prefill):

196.4 tokens/s

quant:

4bit (MLX)

User reports a performance issue with Qwen3.8 27B on an M3 Max 48GB, comparing mlx-serve against LM Studio. The mlx-serve run used the ddalcu/Qwen3.8-27B-MLX-Serve-4bit model at 128k context and achieved 59.6 tok/s decode (54.9-60.2) and 196.4 tok/s prefill, while LM Studio with the Swift-1.5-Qwen3.8-27B-Splash model reached 96.4 tok/s decode (89.1-96.8) and 210.5 tok/s prefill. Both runs used speculative decoding (MTP/draft active). The mlx-serve run showed a 42.1% slowdown under sustained load (59.6 to 34.5 tok/s over 8m 4s), and context scaling was worse than LM Studio at longer contexts (37.2 vs 76.5 tok/s at ~4.2k, 35.6 vs 60.5 at ~8.2k, 34.6 vs 57.6 at ~16.3k). The user reports mlx-serve is 38% slower on decode overall and up to 51% slower at ~4.2k context, while being 18% faster on TTFT.

generation:

91.8 tokens/s

quant:

4-bit (MLX)

User compares Inco Splash 1.1 and oMLX 0.7.0rc1 running an abliterated Qwen3.8-27B on an M3 Max 48GB MacBook Pro. Splash with the DFlash2 draft model reached 91.8 tok/s on code prompts, 52.6 tok/s on creative writing, and 37.7 tok/s on stock Qwen, while oMLX with MTP was in the low-to-mid 30s tok/s. Setup used MLX 4-bit weights on Splash and oQ4e on oMLX, thinking off, temp 0.7, max 600 tokens, median of 3 runs, one engine at a time. User notes the quant is not identical between engines, prompts were short, and long context and concurrent requests were not tested.

generation:

38.0 tokens/s

quant:

T5 (MLX)

User reports Qwen3.8-Flash-Next at ~38 tok/s decode on a 48 GB M3 Max, with prefill at 210-380 tok/s. Setup is the user's oMLX fork with a custom Metal kernel for ternary routed expert gate/up (Bonsai T5 packing, ~1.875 bpw), Q3 expert down projections and the 53 GB n-gram table left on SSD as Q8 and mmapped per token, about 35.6 GiB resident. The oMLX memory guard is on the 'safe' profile with a 48 GB limit, one model loaded and one request at a time. A 65,536-token prompt plus 256 output completed at 30.7 tok/s with a 42.3 GiB physical peak, and 96K trips the prefill guard. Against Unsloth UD-Q4_K_XL, which does not fit in 48 GB, MMLU is 83.0% vs 89.7% and KLD vs Q8_0 is 0.49 vs 0.036.