Local model performance on your hardware

generation:

205.0 tokens/s

quant:

4bit (MLX)

User benchmarks Gemma 4 E2B at 205 t/s decode at 4k context on an Apple M5 Max with 128 GB unified memory. Setup is mlx-vlm 0.4.4 with MLX 0.31.1, 4bit weights, and TurboQuant 2.5-bit KV cache quantization; the model uses 4.7 GB at 4k. Decode falls to 78 t/s at 256k context. TurboQuant gives +15-19% at 128-256k on larger models but is negative on E2B. The same report also covers Gemma 4 E4B, Gemma 4 26B-A4B, Gemma 4 31B, Qwen 3.5 2B and Qwen 3.5 4B.

generation:

47.0 tokens/s

prompt processing (prefill):

854.0 tokens/s

quant:

4bit (MLX)

User reports Qwen3.8-27B at 47.0 t/s generation and 854 t/s prompt processing on an Apple M5 Max with 128 GB unified memory. Setup is mlx-vlm with the 4-bit MLX quantisation and MTP speculative decoding enabled, 15.5 GB resident. The 8-bit quant reaches 28.7 t/s generation and 883 t/s prompt processing at 28.1 GB resident. MTP speculative decoding is reported as worth 2.2x (15.5 t/s to 34.1 t/s at 8-bit). Decode falls to 36.1 t/s at 8K context, and time to first token climbs to about 14 s at that depth.