M5 32GB reports

generation:

40.0 tokens/s

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.6 35B-A3B at about 40 tok/s on an M5 MacBook Pro with 32GB unified memory, against about 17 tok/s for a dense 27B at 4-bit. Setup uses oMLX with an SSD-backed cache server, 8-bit KV cache and context set to roughly 32K. The user says the 16GB dense model plus KV cache filled the 32GB machine and caused a shutdown; the MoE has been stable since.

generation:

57.0 tokens/s

prompt processing (prefill):

1063.0 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-30B-A3B 4-bit at 57 tok/s cold generation on an M5 MacBook Air with 32GB. Setup is MLX with mlx-lm, 4-bit weights, 256 generated tokens, no speculative decoding. Sustained generation drops to 44 tok/s on battery and 39 tok/s on mains; prompt processing of a 4,096-token prompt is 1,063 tok/s cold, 788 sustained on battery and 725 on mains. The decline finishes at about 150 seconds and the rate then holds flat.

generation:

1.0 tokens/s

prompt processing (prefill):

50.0 tokens/s

quant:

4bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at about 50 t/s prefill and about 1 t/s decode on an M5 Air with 32 GB. Setup uses the streamed experts trick with a model of roughly 300B total parameters.

prompt processing (prefill):

3029.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports custom w8a8 kernels on an M5 MacBook Air, with prefill improving from 2193 t/s to 3029 t/s for 130k tokens. User also mentions llama.cpp for Macs.