Local model performance on your hardware

generation:

33.6 tokens/s

prompt processing (prefill):

200.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Code Llama 7B Instruct at 33.6 t/s generation and 200.7 t/s prompt processing on an M2 Pro 32GB Mac mini. Setup is llama.cpp with Metal, Q4_K_M GGUF, one Metal device, and a Japanese Fibonacci-code prompt at --temp 0.1. The figures are native eval phase timers from a single displayed run, not an end-to-end average; context length, layer count, and KV precision are unreported.

generation:

34.8 tokens/s

prompt processing (prefill):

227.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports ALMA-7B-Ja at 34.8 t/s generation and 227.6 t/s prompt processing on an M2 Pro 32GB. Setup is llama.cpp with Metal, Q4_K_M GGUF, run with -n 128; layer count, context length, batch size and KV precision are unreported. The figures are native request phase timers from one displayed run, not an end-to-end average, with no warmup or cache protocol.

generation:

61.7 tokens/s

prompt processing (prefill):

276.8 tokens/s

quant:

4-bit group-64 (MLX)

User reports Qwen3.6 35B-A3B at 61.7 t/s generation and 276.8 t/s prompt processing on a Mac mini with Apple M2 Pro and 32 GB unified memory. Setup is mlx-lm 0.31.3 with MLX 0.31.2, MLX 4-bit group-64 weights of about 19.0 GiB, temperature 0 and --max-tokens 128. The figures are medians of three runs on a 490-token prompt with 128 tokens generated; peak memory was 20.261 GB per run.

generation:

8.6 tokens/s

prompt processing (prefill):

21.9 tokens/s

quant:

IQ4_XS (GGUF)

User reports Qwen3.8 27B IQ4_XS with vision projector on an M2 MacBook Pro 32GB, at 21.9 t/s prompt and 8.6 t/s generation. Setup is llama.cpp built from source, with context set to 128K due to memory constraints. User notes the model is slow on Mac but that thinking quality is good, and mentions previous experience with Qwen3.5 35B-A3B.