Local model performance on your hardware

generation:

63.3 tokens/s

prompt processing (prefill):

700.8 tokens/s

quant:

MXFP4 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports gpt-oss 20B MXFP4 at 63.3 t/s generation and 700.8 t/s prompt processing on a Mac mini M4 Pro with 64 GB unified memory. Setup is llama.cpp build 6280 with MXFP4 GGUF, F16 K/V cache, flash attention, one Metal device, and -ngl 99. The 63.3 t/s is a separate TG128 depth-zero test, not generation after the four large prompts; prompt rates fall from 700.78 t/s at 2048 tokens to 419.74 t/s at 32768.

generation:

70.9 tokens/s

prompt processing (prefill):

1245.6 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Qwen2.5-Omni-3B at 70.9 t/s generation and 1245.6 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q8_0 weights, f16 K/V cache, and one Metal GPU; the run used 512 synthetic prompt positions and 128 serial decode positions. The figures are engine-side loop wall time, not a 512-input/128-output chat request, and no speculative decoding was used.

generation:

54.2 tokens/s

prompt processing (prefill):

957.9 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Qwen3-VL-4B-Instruct at 54.2 t/s generation and 957.9 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA with Q8_0 weights, q8 activations, f16 scales and f16 KV cache, one GPU, 640 positions allocated. The figures are native synthetic counts: 512 prompt positions and 128 serial decode positions, not a single chat request.

generation:

13.0 tokens/s

prompt processing (prefill):

125.6 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

F16

User reports Qwen3.8-27B at 13 t/s generation and 125.6 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with UD-IQ4_XS weights and f16 KV cache, one GPU, 640 positions allocated. The figures are native PP512 and TG128 means over five repetitions, not one 512-input/128-output request.

generation:

13.1 tokens/s

prompt processing (prefill):

126.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3.8-27B at 13.1 t/s generation and 126.2 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU. The figures are native PP512 and TG128 means over five repetitions, not a single chat request; the session allocates 640 positions.

generation:

86.6 tokens/s

prompt processing (prefill):

1545.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Gemma 4 E2B at 86.6 t/s generation and 1545.7 t/s prompt processing on an M4 Pro with 64GB unified memory. Setup is dasLLAMA with Q8_0 weights, f16 KV cache, and one Metal GPU; the session requested 640 positions and no speculative decoding. The figures are native PP512 and TG128 counts over five repetitions, not one 512-in/128-out request.

generation:

17.6 tokens/s

prompt processing (prefill):

143.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Mistral Small 3.1 24B at 17.6 t/s generation and 143.2 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU at 99 layers offloaded. The figures are native PP512 and TG128 counts, not a single 512-input/128-output request; the session requests 640 positions of context.

generation:

94.5 tokens/s

prompt processing (prefill):

806.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3-30B-A3B-Instruct-2507 at 94.5 t/s generation and 806 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 KV cache, and one Metal GPU, with 640 positions allocated. The figures are native PP512 and TG128 means over five repetitions, not a single 512-input/128-output request.

generation:

83.2 tokens/s

prompt processing (prefill):

862.6 tokens/s

quant:

MXFP4 (GGUF)

kv:

F16

User reports gpt-oss-20b at 83.2 t/s generation and 862.6 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with MXFP4 weights, q8 activations, f16 scales and f16 KV cache, one Metal GPU with 99 layers offloaded. Figures are native PP512 and TG128 counts over five repetitions, not a single 512-input/128-output request; the session requests 640 positions of context.

generation:

46.2 tokens/s

prompt processing (prefill):

770.2 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Gemma 4 E4B at 46.2 t/s generation and 770.2 t/s prompt processing on an M4 Pro 64GB. Setup is dasLLAMA with Q8_0 weights, f16 KV cache, and one Metal GPU, with 640 positions allocated. The figures are native PP512 and TG128 counts, not one 512-input/128-output request, and no longer-context performance is inferred.

generation:

30.9 tokens/s

prompt processing (prefill):

298.5 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Gemma 4 12B at 30.9 t/s generation and 298.5 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights and f16 KV cache, one GPU, 640 positions allocated. The figures are native PP512 and TG128 means over five repetitions, not a single 512-input/128-output request; a separate four-stream run reached 34.18746480210349 t/s summed scheduler throughput.

generation:

60.4 tokens/s

prompt processing (prefill):

818.7 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

F16

User reports Gemma 4 26B-A4B-it at 60.4 t/s generation and 818.7 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA 0.6.4 with UD-Q4_K_M weights, f16 KV cache, and one Metal GPU, with 640 positions allocated. The figures are native PP512 and TG128 counts over five repetitions, not a single 512-input/128-output request.

generation:

12.9 tokens/s

prompt processing (prefill):

126.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3.6 27B at 12.9 t/s generation and 126.4 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with Q4_K_M weights, f16 K/V cache, and one Metal GPU; the run used 512 synthetic prompt positions and 128 serial decode positions. The user notes these are native processing counts, not one 512-input/128-output chat request, and that no speculative decoding was requested.

generation:

66.4 tokens/s

prompt processing (prefill):

787.1 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

F16

User reports Qwen3.6-35B-A3B at 66.4 t/s generation and 787.1 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory. Setup is dasLLAMA version 16 with UD-Q4_K_M weights, f16 K/V cache, and one Metal GPU at 640 allocated positions. The figures are native PP512 and TG128 counts, not a single 512-input/128-output request.

generation:

67.5 tokens/s

prompt processing (prefill):

1158.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Qwen2.5-Omni-3B at 67.5 t/s generation and 1158.7 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is llama.cpp commit 6fdd0ac89 with Q8_0 weights, Metal backend, -ngl 99, 10 CPU threads, and f16 K/V cache. The figures are separate native tests from cleared state: PP512 and TG128, each at zero pre-existing depth, not one 512-input/128-output request.

generation:

53.6 tokens/s

prompt processing (prefill):

878.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Qwen3-VL-4B-Instruct at 53.6 t/s generation and 878.7 t/s prompt processing on an M4 Pro 64GB. Setup is llama.cpp commit 6fdd0ac89 with Q8_0 weights, Metal backend, -ngl 99, and 10 CPU threads on one physical GPU. The figures are native llama-bench PP512 and TG128 means over five repetitions at depth 0, not a single 512-input/128-output request.

generation:

12.8 tokens/s

prompt processing (prefill):

126.1 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

F16

User reports Qwen3.8-27B at 12.8 t/s generation and 126.1 t/s prompt processing on an Apple M4 Pro with 64 GB unified memory. Setup is llama.cpp commit 6fdd0ac89 with the UD-IQ4_XS GGUF, Metal backend, -ngl 99, 10 CPU threads, and K/V cache in f16. The figures are native llama-bench means over five repetitions: PP512 at 126.094155 t/s and TG128 at 12.841348 t/s, each starting from cleared state at depth 0, with model load and warmup excluded.

generation:

11.5 tokens/s

prompt processing (prefill):

126.3 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Qwen3.8-27B at 11.5 t/s generation and 126.3 t/s prompt processing on an M4 Pro 64GB. Setup is llama.cpp commit 6fdd0ac89 with Q4_K_M, Metal backend, -ngl 99, 10 CPU threads, and K/V f16. The native figures are PP512 and TG128 from cleared state at depth 0, not a single 512-input/128-output request.

generation:

78.2 tokens/s

prompt processing (prefill):

1558.9 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports Gemma 4 E2B at 78.2 t/s generation and 1558.9 t/s prompt processing on an M4 Pro 64GB. Setup is llama.cpp with Q8_0 weights, Metal backend, -ngl 99, and 10 CPU threads on one physical GPU. The figures are native llama-bench PP512 and TG128 means over five repetitions at depth 0, not a single 512-input/128-output request.

generation:

17.4 tokens/s

prompt processing (prefill):

146.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Mistral Small 3.1 24B at 17.4 t/s generation and 146.8 t/s prompt processing on an M4 Pro with 64 GB unified memory. Setup is llama.cpp commit 6fdd0ac89 with Q4_K_M weights, Metal backend, -ngl 99, 10 CPU threads, and f16 K/V cache. The figures are native llama-bench PP512 and TG128 means over five repetitions at depth 0, not a single chat request.