Local model performance on your hardware

generation:

7.6 tokens/s

prompt processing (prefill):

42.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

User reports Bonsai 2 27B at 7.6 t/s generation and 42.0 t/s prompt processing on an 8GB M2 Mac mini, with all layers offloaded to the GPU and zero swap. Setup is PrismML's llama.cpp fork (prism-b10683-d8f26ee) with a ternary 5.95GB GGUF, flash attention on, -ub 256, q4_0 KV cache, -np 1, and iogpu.wired_limit_mb=6656. The default -ub 512 and f16 KV both fail to allocate; -ub 128 with f16 KV also runs at identical speed. Setting the wired limit to 7168 kernel-panicked the machine. In real use with llama-server at 8k context, a 1,896-token prompt took 47s to first token (pp ~40 t/s) and generated at 7.0 t/s; the same prompt via prefix cache took 0.8s. 16k context works but 32k OOMs, and PQ2_0 (7.21GB) does not fit. The user notes greedy decoding falls into repetition loops and recommends temp 1.0 / top-p 0.95 / top-k 20.

quant:

Q2_K (GGUF)

User benchmarks quantization accuracy damage on Qwen2.5-1.5B and 0.5B across the Q8 to Q2 ladder, running 24,600 generations on an 8GB M2 Air with llama.cpp and MLX. On the 1.5B model, JSON schema conformance stays at 100% at every rung including Q2, GSM8K arithmetic falls from 56.5% to 19.0% with the first significant drop at Q3, and TriviaQA recall falls from 19% to 9%, breaking only at Q2. Q4 significantly beat Q8 on the 1.5B model in both runtimes (p=0.044 llama.cpp, p=0.015 MLX), while on the 0.5B model MLX 4-bit collapses from 45% to 27.5%. Q2_K is strictly dominated on 0.5B: Q3 gives 40.5% at 122 tok/s versus Q2 at 13.5% and 119 tok/s.

generation:

5-6 tokens/s

User reports 31-35 t/s on an M5 MacBook Pro with a custom Swift/Metal engine. Setup is an OpenAI-compatible server with streaming and tool-call support.