- generation:
- 43.5 tokens/s
- quant:
- Q4_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 43.5 t/s generation.
Source evidence (verbatim):
Source: https://antekapetanovic.com/blog/qwen3.5-apple-silicon-benchmark/
# Ollama vs. llama.cpp vs. MLX with Qwen3.5 35B on Apple Silicon
Apple M4 Max, 128GB unified memory, macOS 26.3. Model Qwen3.5 35B-A3B (MoE). Ollama Q4_K_M, llama.cpp Unsloth Dynamic Q4_K_XL, context 65536 for both, MLX 4-bit group quantization. Engine versions not stated. Generation only, mean of 10 runs, single request, no speculative decoding stated. Engine-reported generation tok/s, thinking off, deterministic params: mlx-py 130.2 +/- 0.8, llama.cpp 71.0 +/- 0.1, Ollama 43.5 +/- 0.4. Client wall-clock: mlx-py 114.9, mlx-http 107.6, llama.cpp 65.8, Ollama 40.5. Prompt processing was not reported.
- generation:
- 15.7 tokens/s
- prompt processing (prefill):
- 255.7 tokens/s
- quant:
- Q8_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 15.7 t/s generation and 255.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V cache, and requested GPU layers 99 on one Metal device.
The headline is depth 0: PP512 255.725 and TG128 15.697 tokens/s, separate 512-token prefill and 128-token generation microbenchmarks rather than an end-to-end conversation. At prefix depth 16384, generation falls to 14.931 t/s.
- generation:
- 48.4 tokens/s
- prompt processing (prefill):
- 880.3 tokens/s
- quant:
- Q8_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 48.4 t/s generation and 880.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, and 99 GPU layers requested on one Metal device.
The headline is depth 0; at prefix depth 4096 generation is 47.519 t/s and at 16384 it is 45.097 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 78.8 tokens/s
- prompt processing (prefill):
- 1567.5 tokens/s
- quant:
- Q8_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-4B at 78.8 t/s generation and 1567.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions.
The headline is depth 0; at prefix depth 4096 generation is 76.142 t/s and at 16384 it is 70.660 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 119.7 tokens/s
- prompt processing (prefill):
- 2237.2 tokens/s
- quant:
- Q8_0 (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports SmolLM3-3B at 119.7 t/s generation and 2237.2 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, requested GPU layers 99, single sequence, five repetitions.
The headline is prefix depth 0; at prefix depth 4096 generation falls to 110.40 t/s and at 16384 to 89.60 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 168.6 tokens/s
- prompt processing (prefill):
- 2142.9 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports SmolLM3 3B at 168.6 t/s generation and 2142.9 t/s prompt processing on an M4 Max 128GB.
Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V cache, single sequence, 99 GPU layers requested.
The headline is depth 0; at prefix depth 4096 generation is 149.85 t/s and at 16384 it is 115.65 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 17.2 tokens/s
- prompt processing (prefill):
- 246.3 tokens/s
- quant:
- UD-Q6_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 17.2 t/s generation and 246.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q6_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence.
Figures are depth-0 llama-bench PP512 and TG128 microbenchmarks, not end-to-end; at prefix depths 4096 and 16384 generation falls to 16.94 and 16.29 t/s.
- generation:
- 19.7 tokens/s
- prompt processing (prefill):
- 233.5 tokens/s
- quant:
- UD-Q5_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 19.7 t/s generation and 233.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q5_K_XL GGUF, F16 K/V cache, flash attention, and requested GPU layers 99 on one Metal device.
The headline is depth 0; at prefix depth 4096 generation is 19.45 t/s and at 16384 it is 18.54 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 21.8 tokens/s
- prompt processing (prefill):
- 236.6 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 21.8 t/s generation and 236.6 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q4_K_XL GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on one Metal device.
Figures are depth-0 llama-bench microbenchmarks: PP512 at 236.56 t/s and TG128 at 21.76 t/s, single sequence, five repetitions. At prefix depths 4096 and 16384, generation falls to 21.52 and 20.41 t/s.
- generation:
- 23.5 tokens/s
- prompt processing (prefill):
- 249.0 tokens/s
- quant:
- UD-Q3_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 23.5 t/s generation and 249 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q3_K_XL GGUF, flash attention, F16 K/V, and 99 GPU layers requested on a single Metal device.
The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.07 and 21.77 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 24.8 tokens/s
- prompt processing (prefill):
- 251.1 tokens/s
- quant:
- UD-Q2_K_XL (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 24.8 t/s generation and 251.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q2_K_XL GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions.
Headline is depth 0: PP512 251.10 and TG128 24.76 tokens/s. At prefix depth 4096 generation falls to 24.23 t/s and at 16384 to 22.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 24.2 tokens/s
- prompt processing (prefill):
- 250.4 tokens/s
- quant:
- UD-IQ3_XXS (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 24.2 t/s generation and 250.4 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-IQ3_XXS GGUF, F16 K/V cache, flash attention, 99 GPU layers requested, single sequence.
Headline is depth 0; at prefix depth 4096 generation is 23.81 t/s and at 16384 it is 22.34 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 24.2 tokens/s
- prompt processing (prefill):
- 251.7 tokens/s
- quant:
- UD-IQ2_XXS (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 24.2 t/s generation and 251.7 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-IQ2_XXS GGUF, F16 K/V cache, flash attention, and 99 GPU layers requested on a single Metal device.
The headline is depth 0; at prefix depths 4096 and 16384 generation falls to 23.79 and 22.40 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 81.1 tokens/s
- prompt processing (prefill):
- 1423.5 tokens/s
- quant:
- UD-Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 81.1 t/s generation and 1423.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with UD-Q4_K_M GGUF, flash attention, F16 K/V, single sequence, 99 GPU layers requested.
Figures are depth-0 llama-bench microbenchmarks: PP512 and TG128 separately, not end-to-end. At prefix depth 4096 generation falls to 77.79 t/s and at 16384 to 73.18 t/s.
- generation:
- 68.0 tokens/s
- prompt processing (prefill):
- 832.1 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 68 t/s generation and 832.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory.
Setup is llama.cpp b10794 with the Q4_K_M GGUF, flash attention, F16 K/V cache, requested GPU layers 99, single sequence, five repetitions.
The headline is depth 0; at prefix depth 4096 generation is 66.41 t/s and at 16384 it is 61.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 101.4 tokens/s
- prompt processing (prefill):
- 1476.8 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-4B at 101.4 t/s generation and 1476.8 t/s prompt processing on an M4 Max 128GB.
Setup is llama.cpp b10794 with Q4_K_M and F16 K/V cache, single sequence, 99 GPU layers requested, flash attention.
The headline is depth 0; at prefix depth 4096 generation is 97.37 t/s and at 16384 it is 88.73 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.
- generation:
- 93.6 tokens/s
- prompt processing (prefill):
- 1301.4 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
- flash attention:
- on
- mtp (multi-token prediction):
- off
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Nemotron 3 Nano at 93.6 t/s generation and 1301.4 t/s prompt processing on an M4 Max 128GB.
Setup is llama.cpp with Q4_K_M, flash attention and F16 K/V, single sequence, 99 GPU layers requested.
Figures are separate 512-token prefill and 128-token generation microbenchmarks at prefix depth 0, not an end-to-end run; at depth 4096 generation falls to 91.56 t/s and at 16384 to 87.96 t/s.