Local model performance on your hardware

generation:

93.8 tokens/s

prompt processing (prefill):

1207.0 tokens/s

quant:

4-bit

User benchmarks oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target plus DFlash 2 draft) on a MacBook Pro M5 Max with 48 GB, using a Python code context and 128 generated tokens per test. At a 4096-token prompt the single request reaches 93.8 t/s decode and 1207.0 t/s prefill; longer prompts give 189.4 t/s decode at 16382 tokens and 110.2 t/s at 65536 tokens. Batched continuous batching reaches 413.7 t/s aggregate at 2x and 407.0 t/s at 4x, with peak memory between 19.1 GB and 22.2 GB.

generation:

26.3 tokens/s

prompt processing (prefill):

685.4 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports QwQ-32B at 26.3 t/s generation and 685.4 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention enabled, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

124.3 tokens/s

prompt processing (prefill):

3342.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.6 35B-A3B at 124.3 t/s generation and 3342.7 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

86.5 tokens/s

prompt processing (prefill):

2369.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.5 9B at 86.5 t/s generation and 2369.3 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

135.6 tokens/s

prompt processing (prefill):

4099.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.5 4B at 135.6 t/s generation and 4099.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

111.2 tokens/s

prompt processing (prefill):

3378.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.5-35B-A3B at 111.2 t/s generation and 3378.3 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

29.2 tokens/s

prompt processing (prefill):

730.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen3.5-27B at 29.2 t/s generation and 730.5 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention enabled and n_gpu_layers=99; the context length is unspecified. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput.

generation:

97.7 tokens/s

prompt processing (prefill):

2744.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 8B at 97.7 t/s generation and 2744.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

159.7 tokens/s

prompt processing (prefill):

4972.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 4B at 159.7 t/s generation and 4972.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

26.4 tokens/s

prompt processing (prefill):

667.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 32B at 26.4 t/s generation and 667 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128=26.38 t/s and pp512=667.0 t/s, not one 512-input/128-output request; context length is unspecified.

generation:

148.5 tokens/s

prompt processing (prefill):

3496.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 30B A3B at 148.5 t/s generation and 3496.9 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

57.4 tokens/s

prompt processing (prefill):

1584.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 14B at 57.4 t/s generation and 1584.3 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

288.6 tokens/s

prompt processing (prefill):

11493.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 1.7B at 288.6 t/s generation and 11493.01 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

457.1 tokens/s

prompt processing (prefill):

23288.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 3 0.6B at 457.1 t/s generation and 23288.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

106.0 tokens/s

prompt processing (prefill):

2901.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 2.5 Coder 7B at 106 t/s generation on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

26.3 tokens/s

prompt processing (prefill):

685.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 2.5 Coder 32B at 26.3 t/s generation and 685 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

56.0 tokens/s

prompt processing (prefill):

1546.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Qwen 2.5 Coder 14B at 56.0 t/s generation and 1545.95 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

56.8 tokens/s

prompt processing (prefill):

1537.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Phi-4 at 56.8 t/s generation and 1537.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

164.9 tokens/s

prompt processing (prefill):

5412.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Phi-4 at 164.9 t/s generation and 5412.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

165.0 tokens/s

prompt processing (prefill):

5416.2 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

User reports Phi-4-mini Q4_K_M at 165 t/s generation and 5416.2 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, n_gpu_layers=99 and flash attention enabled, run as separate llama-bench microtests. The figures are native tg128 and pp512 tests, not one 512-input/128-output request or server concurrency; context length is unspecified.