- generation:
- 115.4 tokens/s
- prompt processing (prefill):
- 3225.7 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.1 8B at 115.4 t/s generation and 3225.7 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one model per process.
The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1932.3, pp256=2487.89 and tg256=115.52 t/s.
- generation:
- 237.9 tokens/s
- prompt processing (prefill):
- 7307.2 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 3B Instruct 4-bit at 237.9 t/s generation and 7307.2 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit Safetensors weights, batch size 1, unquantized KV cache and no speculative decoding.
The figures are arithmetic means of five trials at prompt 512 and generation 128; the benchmark is synthetic token throughput, not end-to-end request throughput.
- generation:
- 527.7 tokens/s
- prompt processing (prefill):
- 17477.0 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.2 1B Instruct 4-bit at 527.7 t/s generation and 17477 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights and unquantized KV cache, batch size 1, one warmup plus five measured trials.
The headline uses pp512 and tg128, not a best-of sweep; the source also lists pp128 7439.85, pp256 13408.64 and tg256 542.87 t/s.
- generation:
- 121.3 tokens/s
- prompt processing (prefill):
- 3351.9 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 121.3 t/s generation and 3351.9 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache, one Mac Studio with 18 CPU cores and 40 GPU cores.
The headline uses pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also lists pp128=2139.57, pp256=2956.62, tg256=121.99 t/s.
- generation:
- 110.6 tokens/s
- prompt processing (prefill):
- 3150.6 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-8B-4bit at 110.6 t/s generation and 3150.6 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX via mlx-lm 0.31.3 with 4-bit weights and unquantized KV cache, batch size 1, one model per process.
The headline figures are pp512 and tg128 from a five-trial mean, not a best-of sweep; the source also records pp128=1876.59, pp256=2405.12 and tg256=111.03 t/s.
- generation:
- 28.3 tokens/s
- prompt processing (prefill):
- 731.4 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 28.3 t/s generation and 731.4 t/s prompt processing on an M5 Max with 48GB unified RAM.
Setup is MLX with 4-bit weights and unquantized KV cache, one model invocation per process, batch size 1, no speculative decoding.
The headline figures are pp512 and tg128, not a best-of sweep or average; the source also lists pp128=493.38, pp256=616.57 and tg256=28.44 t/s. Peak MLX allocator memory was 19.119 GB.
- generation:
- 60.7 tokens/s
- prompt processing (prefill):
- 1714.8 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek R1 Distill at 60.7 t/s generation and 1714.8 t/s prompt processing on an M5 Max 48GB.
Setup is MLX with 4-bit weights, batch size 1, unquantized KV cache, one model invocation per process.
The headline uses pp512 and tg128, not a best-of sweep or average; the benchmark is synthetic text-token throughput, not end-to-end request throughput.