Local model performance on your hardware

generation:

8.0 tokens/s

quant:

F16 (GGUF)

mtp (multi-token prediction):

on

User benchmarks Qwen 3.6 27B F16 on an M2 Max 96GB with llama.cpp and MTP speculative decoding, reporting 8-18 tok/s depending on the task. Without MTP the same setup reaches 6.6 tok/s. A Q8 quant was also tested but performed worse. Context up to 150k+ tokens is usable. The user notes chat template fixes were critical, and the setup was used for agentic coding to create a Pacman game.

generation:

28.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q4

User reports MTP speculative decoding gives a 2.5x speedup on an M2 Max 96GB. Setup is Q5_K_M quant with a q4_0 KV cache. The user also provides hardware recommendations for various Apple Silicon and NVIDIA GPUs.