Local model performance on your hardware

generation:

159.7 tokens/s

quant:

4-bit (MLX)

User reports Nemotron-3 Nano 30B-A3B at 159.7 t/s decode on an M4 Max with 128 GB, batch 1 and greedy, measured with mlx-lm 0.31.3 on mlx 0.32.2. The model was the mlx-community 4-bit build at 17.8 GB; the figure is mlx_lm generate's own Generation rate over 256 tokens, taken as the median of three processes. The post compares mlx-lm against llama.cpp b8680 on the same machine for hybrid Mamba-2 models, but only the MLX row for Nemotron-3 Nano is given here.

generation:

57.0 tokens/s

quant:

4bpw-sh (EXL3)

User publishes two EXL3 quants of Qwen3.8-Flash-Next for the sushi engine on Apple Silicon, reporting about 57 tok/s decode for the 4-bit pack and 63 tok/s for the 5-bit on an M4 Max with 128 GB. Routed experts come from ExLlamaV3 v1.6.0; dense layers, embeddings, vision tower and MTP head are copied from beamster's Sushi-4bpw. Prefill at 64K is about 690 tok/s. The 4-bit pack is 63.7 GiB and the 5-bit 77.8 GiB; KLD against BF16 is 0.94 for 4-bit and 0.79 for 5-bit. Needles in a 400K-token document scored 4/4 for both, and tool calling 28/28 for the 5-bit. The eval is 24 prompts on one machine.

generation:

43.0 tokens/s

User reports Qwen3.8 27B (abliterated) at ~43 tok/s on an M4 Max 128GB. Setup is mlx-serve 26.10.2 with MTP enabled; the 27B build is 16GB. The user also lists other abliterated models in their picker: Smart 35B at ~106 tok/s (24GB), Vision 27B at ~22 tok/s (15GB), Coder 80B at ~78 tok/s (49GB), DeepSeek-V4-Flash 284B at ~26 tok/s (75GB), and Flash-Next 125B at ~90 tok/s (77GB). They ask for suggestions on better packs and MTP or mlx-serve settings for this hardware.

generation:

43.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports 43.5 t/s generation. Source evidence (verbatim): Source: https://antekapetanovic.com/blog/qwen3.5-apple-silicon-benchmark/ # Ollama vs. llama.cpp vs. MLX with Qwen3.5 35B on Apple Silicon Apple M4 Max, 128GB unified memory, macOS 26.3. Model Qwen3.5 35B-A3B (MoE). Ollama Q4_K_M, llama.cpp Unsloth Dynamic Q4_K_XL, context 65536 for both, MLX 4-bit group quantization. Engine versions not stated. Generation only, mean of 10 runs, single request, no speculative decoding stated. Engine-reported generation tok/s, thinking off, deterministic params: mlx-py 130.2 +/- 0.8, llama.cpp 71.0 +/- 0.1, Ollama 43.5 +/- 0.4. Client wall-clock: mlx-py 114.9, mlx-http 107.6, llama.cpp 65.8, Ollama 40.5. Prompt processing was not reported.

generation:

15.7 tokens/s

prompt processing (prefill):

255.7 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 15.7 t/s generation and 255.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V cache, and requested GPU layers 99 on one Metal device. The headline is depth 0: PP512 255.725 and TG128 15.697 tokens/s, separate 512-token prefill and 128-token generation microbenchmarks rather than an end-to-end conversation. At prefix depth 16384, generation falls to 14.931 t/s.

generation:

48.4 tokens/s

prompt processing (prefill):

880.3 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-9B at 48.4 t/s generation and 880.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, and 99 GPU layers requested on one Metal device. The headline is depth 0; at prefix depth 4096 generation is 47.519 t/s and at 16384 it is 45.097 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

78.8 tokens/s

prompt processing (prefill):

1567.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-4B at 78.8 t/s generation and 1567.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 76.142 t/s and at 16384 it is 70.660 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

119.7 tokens/s

prompt processing (prefill):

2237.2 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports SmolLM3-3B at 119.7 t/s generation and 2237.2 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, requested GPU layers 99, single sequence, five repetitions. The headline is prefix depth 0; at prefix depth 4096 generation falls to 110.40 t/s and at 16384 to 89.60 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

168.6 tokens/s

prompt processing (prefill):

2142.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports SmolLM3 3B at 168.6 t/s generation and 2142.9 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V cache, single sequence, 99 GPU layers requested. The headline is depth 0; at prefix depth 4096 generation is 149.85 t/s and at 16384 it is 115.65 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

17.2 tokens/s

prompt processing (prefill):

246.3 tokens/s

quant:

UD-Q6_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 17.2 t/s generation and 246.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q6_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence. Figures are depth-0 llama-bench PP512 and TG128 microbenchmarks, not end-to-end; at prefix depths 4096 and 16384 generation falls to 16.94 and 16.29 t/s.

generation:

19.7 tokens/s

prompt processing (prefill):

233.5 tokens/s

quant:

UD-Q5_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 19.7 t/s generation and 233.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q5_K_XL GGUF, F16 K/V cache, flash attention, and requested GPU layers 99 on one Metal device. The headline is depth 0; at prefix depth 4096 generation is 19.45 t/s and at 16384 it is 18.54 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

21.8 tokens/s

prompt processing (prefill):

236.6 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 21.8 t/s generation and 236.6 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_XL GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on one Metal device. Figures are depth-0 llama-bench microbenchmarks: PP512 at 236.56 t/s and TG128 at 21.76 t/s, single sequence, five repetitions. At prefix depths 4096 and 16384, generation falls to 21.52 and 20.41 t/s.

generation:

23.5 tokens/s

prompt processing (prefill):

249.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 23.5 t/s generation and 249 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q3_K_XL GGUF, flash attention, F16 K/V, and 99 GPU layers requested on a single Metal device. The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.07 and 21.77 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.8 tokens/s

prompt processing (prefill):

251.1 tokens/s

quant:

UD-Q2_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 24.8 t/s generation and 251.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q2_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions. Headline is depth 0: PP512 251.10 and TG128 24.76 tokens/s. At prefix depth 4096 generation falls to 24.23 t/s and at 16384 to 22.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.2 tokens/s

prompt processing (prefill):

250.4 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 24.2 t/s generation and 250.4 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-IQ3_XXS GGUF, F16 K/V cache, flash attention, 99 GPU layers requested, single sequence. Headline is depth 0; at prefix depth 4096 generation is 23.81 t/s and at 16384 it is 22.34 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

24.2 tokens/s

prompt processing (prefill):

251.7 tokens/s

quant:

UD-IQ2_XXS (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 24.2 t/s generation and 251.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-IQ2_XXS GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on a single Metal device. The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.79 and 22.40 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

81.1 tokens/s

prompt processing (prefill):

1423.5 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.6-35B-A3B at 81.1 t/s generation and 1423.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_M GGUF, flash attention, F16 K/V, single sequence, 99 GPU layers requested. Figures are depth-0 llama-bench microbenchmarks: PP512 and TG128 separately, not end-to-end. At prefix depth 4096 generation falls to 77.79 t/s and at 16384 to 73.18 t/s.

generation:

68.0 tokens/s

prompt processing (prefill):

832.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-9B at 68 t/s generation and 832.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with the Q4_K_M GGUF, flash attention, F16 K/V cache, requested GPU layers 99, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 66.41 t/s and at 16384 it is 61.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

101.4 tokens/s

prompt processing (prefill):

1476.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-4B at 101.4 t/s generation and 1476.8 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp b10794 with Q4_K_M and F16 K/V cache, single sequence, 99 GPU layers requested, flash attention. The headline is depth 0; at prefix depth 4096 generation is 97.37 t/s and at 16384 it is 88.73 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

93.6 tokens/s

prompt processing (prefill):

1301.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Nemotron 3 Nano at 93.6 t/s generation and 1301.4 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V, single sequence, 99 GPU layers requested. Figures are separate 512-token prefill and 128-token generation microbenchmarks at prefix depth 0, not an end-to-end run; at depth 4096 generation falls to 91.56 t/s and at 16384 to 87.96 t/s.