M5 16GB reports

generation:

33-37 tokens/s

quant:

qvex-q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B running at about 25 tok/s without speculation and 33-37 tok/s with MTP speculative decoding on a fanless MacBook Air M5 16GB. Setup is a custom MLX engine (QVex-3.5) with a self-made qvex-q4 4-bit imatrix quant, using Qwen's own MTP head as the draft model (about 490 MB). The custom quant is 2.7x closer to bf16 than stock MLX 4-bit (KL 0.026 vs 0.069). A 3-bit build was about 13% faster but used 20-45% more energy per token, and MTP costs 10-25% more energy per token. With a 512-token thinking budget the model solved 24/24 tasks; the user calls this a sanity check rather than a benchmark, tested on one machine only.

generation:

17.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 35B A3B running at roughly 17 t/s on a MacBook Pro M5 16GB. Setup uses slotstream; no engine, quant or context length stated. The figure is given as approximate.

generation:

9.0 tokens/s

quant:

Q3_xxs

kv:

8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether about 9 t/s is normal for their setup, and is new to local LLMs. User also asks for model recommendations for PDF summaries on 16 GB of RAM.