Local model performance on your hardware

generation:

19.7 tokens/s

prompt processing (prefill):

223.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 8B at 19.7 t/s generation and 223.8 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the GPU, four CPU threads. The figures are separate llama-bench tests (pp512 and tg128), each the mean of three repetitions at zero initial depth, not one 512-input/128-output request; at 8192 prefill depth generation is 16.1 t/s.

generation:

36.2 tokens/s

prompt processing (prefill):

416.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 4B at 36.2 t/s generation and 416.6 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the 10-core integrated GPU. The figures are separate llama-bench tests, pp512 and tg128, each the mean of three repetitions at zero initial depth, not one 512-input/128-output request.