Local model performance on your hardware

generation:

24.1 tokens/s

prompt processing (prefill):

221.3 tokens/s

quant:

Q4_0 (GGUF)

User reports LLaMA-family 7B at 24.1 t/s generation and 221.3 t/s prompt processing on a Mac mini M4 with 32 GB unified RAM. Setup is llama.cpp build 1550 with Q4_0 and F16 KV cache, Metal backend at ngl 99 on one GPU. The figures are llama-bench PP512 and TG128 synthetic tests, not a 512-token prompt followed by a 128-token answer; exact source rates are 221.29 and 24.11 t/s.

generation:

13.5 tokens/s

prompt processing (prefill):

223.6 tokens/s

quant:

Q8_0 (GGUF)

User reports LLaMA-family 7B mostly Q8_0 at 13.5 t/s generation and 223.6 t/s prompt processing on a Mac mini M4 with 32 GB unified RAM. Setup is llama.cpp build 1550 with Q8_0 and F16 KV cache, Metal backend at ngl 99, one GPU. The figures are llama-bench PP512 and TG128, independent synthetic prompt and generation tests rather than a 512-token prompt followed by a 128-token answer.

generation:

7.4 tokens/s

prompt processing (prefill):

230.2 tokens/s

quant:

F16 (GGUF)

User reports a LLaMA-family 7B model at 7.4 t/s generation and 230.2 t/s prompt processing on a Mac mini M4 with 32 GB unified RAM. Setup is llama.cpp build 1550 with F16 quantization and F16 KV cache, Metal backend at ngl 99, one GPU. The figures are llama-bench PP512 and TG128 results, independent synthetic prompt and generation tests rather than a 512-token prompt followed by a 128-token answer; exact source rates are 230.18 ± 0.16 t/s and 7.43 ± 0.02 t/s.

generation:

19.7 tokens/s

prompt processing (prefill):

223.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 8B at 19.7 t/s generation and 223.8 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the GPU, four CPU threads. The figures are separate llama-bench tests (pp512 and tg128), each the mean of three repetitions at zero initial depth, not one 512-input/128-output request; at 8192 prefill depth generation is 16.1 t/s.

generation:

36.2 tokens/s

prompt processing (prefill):

416.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 4B at 36.2 t/s generation and 416.6 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the 10-core integrated GPU. The figures are separate llama-bench tests, pp512 and tg128, each the mean of three repetitions at zero initial depth, not one 512-input/128-output request.

generation:

5.3 tokens/s

prompt processing (prefill):

90.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 31B at 5.3 t/s generation and 90 t/s prompt processing on an M4 32GB Mac mini. Setup is Ollama 0.30.5 with GGUF Q4_K_M and num_ctx=4096, 100% GPU offload, 20.29 GB loaded memory including KV cache. The figures are three-run averages from a fixed Chinese long-form prompt; 4096 is the configured context, not measured input length, and per-run counters are unpublished.

generation:

32.0 tokens/s

prompt processing (prefill):

657.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 26B-A4B at 32 t/s generation and 657 t/s prompt processing on an M4 32GB Mac mini. Setup is Ollama 0.30.5 with GGUF Q4_K_M and num_ctx=4096, 100% GPU offload, 17.37 GB loaded memory including KV cache. Both figures are three-run averages; the 400-character prompt instruction is not an output-token count, and 4096 is the configured context, not measured input length.

generation:

12.5 tokens/s

prompt processing (prefill):

236.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 12B at 12.5 t/s generation and 236 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory. Setup is Ollama 0.30.5 with GGUF Q4_K_M and num_ctx=4096, reporting 100% GPU offload and 8.05 GB loaded memory including KV cache. Both figures are three-run averages; the 4096 is the configured context, not measured input length, and the prompt asks for at least 400 Chinese characters.

generation:

29.9 tokens/s

prompt processing (prefill):

566.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E4B at 29.9 t/s generation and 566 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory. Setup is Ollama 0.30.5 with GGUF Q4_K_M at num_ctx=4096, with 100% GPU offload and 9.55 GB loaded memory including KV cache. The figures are source-reported three-run averages, with decode from eval_count/eval_duration and prompt processing from prompt_eval_count/prompt_eval_duration, excluding client and model-loading overhead.

generation:

55.0 tokens/s

prompt processing (prefill):

1089.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E2B at 55 t/s generation and 1089 t/s prompt processing on an Apple Mac mini M4 with 32 GB unified memory. Setup is Ollama 0.30.5 with GGUF Q4_K_M at num_ctx=4096 and 100% GPU offload, using the Ollama library gemma4:e2b tag. The figures are a three-run average (54.97 t/s decode, 1089 t/s prompt) from a fixed Chinese long-form prompt; reported loaded memory is 7.06 GB including KV cache.

generation:

8.7 tokens/s

prompt processing (prefill):

110.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

flash attention:

on

User reports Gemma 4 12B at 8.7 t/s generation and 110.4 t/s prompt processing on an M4 32GB Mac mini. Setup is Ollama 0.30.5 with Q4_K_M and q8_0 KV cache, num_ctx=8192, flash attention enabled, one card. The row is the first long-context tier: 5713 actual input tokens and 512 actual output tokens, not a full 8192-token prompt or an average. The source TTFT estimate is 64.5 s and request wall time 123.4 s.

generation:

16.7 tokens/s

prompt processing (prefill):

245.5 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

flash attention:

on

User reports Gemma 4 E4B at 16.7 t/s generation and 245.5 t/s prompt processing on an M4 32GB Mac mini. Setup is Ollama 0.30.5 with Q4_K_M and q8_0 KV cache at 8192 context, one Mac mini M4 with 32 GB unified memory. The figure is a single long-context row with 5709 actual input and 505 actual output tokens, not an average or a full 8192-token prompt; the source TTFT estimate is 23.53 s.

generation:

8.0 tokens/s

User asks about an MTP or DFlash head for Qwen 3.8 27B, and mentions 35B-A3B as a daily driver. User reports about 8 tok/s for Qwen 3.6 27B with MTP on 32 GB of unified memory.