Mac M4 reports

generation:

22.6 tokens/s

prompt processing (prefill):

239.7 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 2.5 Coder 7B at 22.6 t/s generation and 239.7 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, full Metal GPU offload, 4 CPU threads, and F16 K/V cache. The figures are separate synthetic tests, PP512 and TG128, each averaged over three repetitions; the native rates were 239.73 +/- 2.07 t/s and 22.60 +/- 0.01 t/s.

generation:

46.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 46 t/s generation and 530 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, Metal backend, full GPU offload, F16 K/V cache, and four CPU threads. The figures are separate synthetic tests: PP512 at 530.01 +/- 0.86 t/s and TG128 at 45.95 +/- 0.02 t/s, three repetitions each, not a paired prompt-and-answer run.

generation:

15.7 tokens/s

prompt processing (prefill):

255.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 15.7 t/s generation and 255.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V cache, and requested GPU layers 99 on one Metal device. The headline is depth 0: PP512 255.725 and TG128 15.697 tokens/s, separate 512-token prefill and 128-token generation microbenchmarks rather than an end-to-end conversation. At prefix depth 16384, generation falls to 14.931 t/s.

generation:

48.4 tokens/s

prompt processing (prefill):

880.3 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 48.4 t/s generation and 880.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, and 99 GPU layers requested on one Metal device. The headline is depth 0; at prefix depth 4096 generation is 47.519 t/s and at 16384 it is 45.097 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

78.8 tokens/s

prompt processing (prefill):

1567.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 78.8 t/s generation and 1567.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 76.142 t/s and at 16384 it is 70.660 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

119.7 tokens/s

prompt processing (prefill):

2237.2 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports SmolLM3-3B at 119.7 t/s generation and 2237.2 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, requested GPU layers 99, single sequence, five repetitions. The headline is prefix depth 0; at prefix depth 4096 generation falls to 110.40 t/s and at 16384 to 89.60 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

168.6 tokens/s

prompt processing (prefill):

2142.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports SmolLM3 3B at 168.6 t/s generation and 2142.9 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V cache, single sequence, 99 GPU layers requested. The headline is depth 0; at prefix depth 4096 generation is 149.85 t/s and at 16384 it is 115.65 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

17.2 tokens/s

prompt processing (prefill):

246.3 tokens/s

quant:

UD-Q6_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 17.2 t/s generation and 246.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q6_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence. Figures are depth-0 llama-bench PP512 and TG128 microbenchmarks, not end-to-end; at prefix depths 4096 and 16384 generation falls to 16.94 and 16.29 t/s.

generation:

19.7 tokens/s

prompt processing (prefill):

233.5 tokens/s

quant:

UD-Q5_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 19.7 t/s generation and 233.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q5_K_XL GGUF, F16 K/V cache, flash attention, and requested GPU layers 99 on one Metal device. The headline is depth 0; at prefix depth 4096 generation is 19.45 t/s and at 16384 it is 18.54 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

21.8 tokens/s

prompt processing (prefill):

236.6 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 21.8 t/s generation and 236.6 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_XL GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on one Metal device. Figures are depth-0 llama-bench microbenchmarks: PP512 at 236.56 t/s and TG128 at 21.76 t/s, single sequence, five repetitions. At prefix depths 4096 and 16384, generation falls to 21.52 and 20.41 t/s.

generation:

23.5 tokens/s

prompt processing (prefill):

249.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 23.5 t/s generation and 249 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q3_K_XL GGUF, flash attention, F16 K/V, and 99 GPU layers requested on a single Metal device. The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.07 and 21.77 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.8 tokens/s

prompt processing (prefill):

251.1 tokens/s

quant:

UD-Q2_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 24.8 t/s generation and 251.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q2_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions. Headline is depth 0: PP512 251.10 and TG128 24.76 tokens/s. At prefix depth 4096 generation falls to 24.23 t/s and at 16384 to 22.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.2 tokens/s

prompt processing (prefill):

250.4 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 24.2 t/s generation and 250.4 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-IQ3_XXS GGUF, F16 K/V cache, flash attention, 99 GPU layers requested, single sequence. Headline is depth 0; at prefix depth 4096 generation is 23.81 t/s and at 16384 it is 22.34 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.2 tokens/s

prompt processing (prefill):

251.7 tokens/s

quant:

UD-IQ2_XXS (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 24.2 t/s generation and 251.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-IQ2_XXS GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on a single Metal device. The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.79 and 22.40 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

81.1 tokens/s

prompt processing (prefill):

1423.5 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 81.1 t/s generation and 1423.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_M GGUF, flash attention, F16 K/V, single sequence, 99 GPU layers requested. Figures are depth-0 llama-bench microbenchmarks: PP512 and TG128 separately, not end-to-end. At prefix depth 4096 generation falls to 77.79 t/s and at 16384 to 73.18 t/s.

generation:

68.0 tokens/s

prompt processing (prefill):

832.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 68 t/s generation and 832.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with the Q4_K_M GGUF, flash attention, F16 K/V cache, requested GPU layers 99, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 66.41 t/s and at 16384 it is 61.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

101.4 tokens/s

prompt processing (prefill):

1476.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 101.4 t/s generation and 1476.8 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp b10794 with Q4_K_M and F16 K/V cache, single sequence, 99 GPU layers requested, flash attention. The headline is depth 0; at prefix depth 4096 generation is 97.37 t/s and at 16384 it is 88.73 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

93.6 tokens/s

prompt processing (prefill):

1301.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Nemotron 3 Nano at 93.6 t/s generation and 1301.4 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V, single sequence, 99 GPU layers requested. Figures are separate 512-token prefill and 128-token generation microbenchmarks at prefix depth 0, not an end-to-end run; at depth 4096 generation falls to 91.56 t/s and at 16384 to 87.96 t/s.

generation:

91.8 tokens/s

prompt processing (prefill):

1277.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Nemotron 3 Nano at 91.8 t/s generation and 1277.1 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V, single sequence, GPU layers 99. The headline is depth 0; at prefix depth 4096 generation is 90.27 t/s and at 16384 it is 86.07 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

28.5 tokens/s

prompt processing (prefill):

253.9 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Muse-Glimmer-30B at 28.5 t/s generation and 253.9 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_XL and F16 K/V cache, flash attention, 99 GPU layers requested, single sequence. Figures are separate 512-token prefill and 128-token generation microbenchmarks at prefix depth 0; at depth 4096 generation is 27.82 t/s and at 16384 it is 27.07 t/s.