Mac M5 reports

generation:

165.0 tokens/s

prompt processing (prefill):

5416.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Phi-4-mini Q4_K_M at 165 t/s generation and 5416.2 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, n_gpu_layers=99 and flash attention enabled, run as separate llama-bench microtests. The figures are native tg128 and pp512 tests, not one 512-input/128-output request or server concurrency; context length is unspecified.

generation:

56.3 tokens/s

prompt processing (prefill):

1534.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Phi-4 at 56.3 t/s generation and 1534.2 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

37.2 tokens/s

prompt processing (prefill):

977.1 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Mistral-Small-3.1-24B at 37.2 t/s generation and 977.1 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

68.6 tokens/s

prompt processing (prefill):

1872.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Mistral-Nemo-Instruct-2407 at 68.6 t/s generation and 1872.2 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified and KV cache precision was not captured.

generation:

107.1 tokens/s

prompt processing (prefill):

2772.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Mistral-7B-Instruct-v0.3 at 107.1 t/s generation and 2772.2 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

199.9 tokens/s

prompt processing (prefill):

6122.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 199.9 t/s generation and 6122.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

439.6 tokens/s

prompt processing (prefill):

16967.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 1B at 439.6 t/s generation and 16967.9 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

101.1 tokens/s

prompt processing (prefill):

2758.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.1 8B at 101.1 t/s generation and 2758.3 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified and KV cache precision was not captured.

generation:

110.5 tokens/s

prompt processing (prefill):

4217.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 E4B at 110.5 t/s generation and 4217.3 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput.

generation:

173.7 tokens/s

prompt processing (prefill):

7021.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 E2B at 173.7 t/s generation and 7021.2 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; the context field is intentionally unspecified.

generation:

25.2 tokens/s

prompt processing (prefill):

626.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B at 25.2 t/s generation and 626.5 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Q4_K_M weights and Metal, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput.

generation:

116.9 tokens/s

prompt processing (prefill):

3423.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 116.9 t/s generation and 3423.2 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified and KV cache precision was not captured.

generation:

150.5 tokens/s

prompt processing (prefill):

5760.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 3 4B at 150.5 t/s generation and 5760.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

29.9 tokens/s

prompt processing (prefill):

751.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 3 27B at 29.9 t/s generation and 751 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

294.9 tokens/s

prompt processing (prefill):

16750.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 3 1B at 294.9 t/s generation and 16750.2 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99, flash_attention=true and threads=0. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

61.5 tokens/s

prompt processing (prefill):

1721.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 3 12B at 61.5 t/s generation and 1721.5 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

37.6 tokens/s

prompt processing (prefill):

978.1 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Devstral-Small-24B at 37.6 t/s generation and 978.1 t/s prompt processing on a single M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128=37.56 t/s and pp512=978.12 t/s, not one 512-input/128-output request; context length is unspecified.

generation:

105.6 tokens/s

prompt processing (prefill):

2899.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek R1 Distill at 105.6 t/s generation and 2899.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate native llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

26.4 tokens/s

prompt processing (prefill):

685.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek R1 Distill at 26.4 t/s generation and 685.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

56.5 tokens/s

prompt processing (prefill):

1547.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek R1 Distill at 56.5 t/s generation and 1547.5 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128=56.48 t/s and pp512=1547.46 t/s, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.