Qwen3 reports

generation:

116.9 tokens/s

prompt processing (prefill):

942.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 4B 4-bit at 116.9 t/s generation and 942.3 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one request of 512 input tokens and 128 output tokens, greedy temperature 0, no speculative decoding. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.

generation:

19.7 tokens/s

prompt processing (prefill):

223.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 8B at 19.7 t/s generation and 223.8 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the GPU, four CPU threads. The figures are separate llama-bench tests (pp512 and tg128), each the mean of three repetitions at zero initial depth, not one 512-input/128-output request; at 8192 prefill depth generation is 16.1 t/s.

generation:

36.2 tokens/s

prompt processing (prefill):

416.6 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 4B at 36.2 t/s generation and 416.6 t/s prompt processing on a Mac mini M4 with 32 GB unified memory. Setup is llama.cpp b10929 with the Q4_K_M GGUF and Metal, all layers on the 10-core integrated GPU. The figures are separate llama-bench tests, pp512 and tg128, each the mean of three repetitions at zero initial depth, not one 512-input/128-output request.

generation:

80.3 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3-14B at 80.3 t/s on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is LM Studio with 4-bit MLX weights on one machine, 36 CPU cores and 80 GPU cores. The 80.3 t/s is the mean of three runs; TTFT is 3.53 seconds, and the displayed 2,754 prompt tokens/s is prompt-token count divided by TTFT, not native prefill. Configured context capacity is unknown.

generation:

12.7 tokens/s

prompt processing (prefill):

116.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-14B-4bit at 12.7 t/s generation on one Apple Mac mini M4 with 16 GB unified memory. Setup is mlx-lm 0.29.1 native mlx_lm generate with 4-bit MLX safetensors, --temp 0 and --max-tokens 128, no speculative decoding or prompt cache. The separate long run reports 116 t/s prompt processing at 3,796 input tokens; the two short runs both measured 12.7 t/s, and peak memory was 8.4GB.

generation:

23.2 tokens/s

prompt processing (prefill):

219.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-8B-4bit at 23.2 t/s generation on a single Apple Mac mini M4 with 16 GB unified memory. Setup is MLX via mlx-lm 0.29.1 native mlx_lm generate, 4-bit, with --temp 0 and --max-tokens 128; no speculative decoding or prompt cache. A separate long request reports 219 t/s prompt processing at 3,796 input tokens, so the two figures come from different requests. Peak memory is reported at 4.7GB.

generation:

97.7 tokens/s

prompt processing (prefill):

2744.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 8B at 97.7 t/s generation and 2744.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

159.7 tokens/s

prompt processing (prefill):

4972.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 4B at 159.7 t/s generation and 4972.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.

generation:

26.4 tokens/s

prompt processing (prefill):

667.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 32B at 26.4 t/s generation and 667 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99. The figures are separate llama-bench microtests, tg128=26.38 t/s and pp512=667.0 t/s, not one 512-input/128-output request; context length is unspecified.

generation:

148.5 tokens/s

prompt processing (prefill):

3496.9 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 30B A3B at 148.5 t/s generation and 3496.9 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

57.4 tokens/s

prompt processing (prefill):

1584.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 14B at 57.4 t/s generation and 1584.3 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

288.6 tokens/s

prompt processing (prefill):

11493.0 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 1.7B at 288.6 t/s generation and 11493.01 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

457.1 tokens/s

prompt processing (prefill):

23288.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 0.6B at 457.1 t/s generation and 23288.5 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

186.0 tokens/s

prompt processing (prefill):

5505.7 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 4B 4-bit at 186 t/s generation and 5505.7 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 48 GB unified RAM. The headline figures are pp512=5505.67 and tg128=185.98 tokens/s, not a best-of sweep or an average across prompt lengths; the benchmark is synthetic text-token throughput with batch size 1.

generation:

28.4 tokens/s

prompt processing (prefill):

730.9 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-32B-4bit at 28.4 t/s generation and 730.9 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX via mlx-lm 0.31.3 with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials. The headline figures are pp512 and tg128, not a best-of sweep or average; peak MLX allocator memory was 19.151 GB.

generation:

147.2 tokens/s

prompt processing (prefill):

3458.2 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 30B-A3B 4-bit at 147.2 t/s generation and 3458.2 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 48 GB unified RAM. The figures are single-batch synthetic token rates, not end-to-end request throughput; peak MLX allocator memory was 17.755 GB.

generation:

62.6 tokens/s

prompt processing (prefill):

1738.6 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-14B-4bit at 62.6 t/s generation and 1738.6 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process. The stored figures are the pp512 and tg128 values from a five-trial mean; the source also records pp128=1145.62, pp256=1393.0, tg256=62.71 t/s.

generation:

366.0 tokens/s

prompt processing (prefill):

12362.2 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 1.7B at 366 t/s generation and 12362.2 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio with 18 CPU cores and 40 GPU cores. The figures are native engine phase rates at batch size 1, not end-to-end request throughput; the source lists tg128=365.99 and pp512=12362.19.

generation:

570.6 tokens/s

prompt processing (prefill):

22693.8 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3 0.6B 4-bit at 570.6 t/s generation and 22693.8 t/s prompt processing on an M5 Max 48GB. Setup is MLX with 4-bit weights and unquantized KV cache on one Mac Studio, batch size 1, no speculative decoding. The headline figures are the pp512 and tg128 rates, not a best-of sweep or average; the source also lists pp128=9042.7, pp256=12175.94 and tg256=594.11 t/s.

generation:

110.6 tokens/s

prompt processing (prefill):

3150.6 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-8B-4bit at 110.6 t/s generation and 3150.6 t/s prompt processing on an M5 Max with 48GB unified RAM. Setup is MLX via mlx-lm 0.31.3 with 4-bit weights and unquantized KV cache, batch size 1, one model per process. The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1876.59, pp256=2405.12 and tg256=111.03 t/s.