Mac M1 reports

generation:

47.7 tokens/s

prompt processing (prefill):

456.1 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 47.7 t/s generation and 456.1 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with IQ3_S quant, Metal backend and ngl 99, one participating GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

47.7 tokens/s

prompt processing (prefill):

459.5 tokens/s

quant:

IQ3_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 47.7 t/s generation and 459.5 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with IQ3_M and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

44.2 tokens/s

prompt processing (prefill):

469.6 tokens/s

quant:

Q3_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 44.2 t/s generation and 469.6 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q3_K_M and Metal, ngl 99, on one GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

40.9 tokens/s

prompt processing (prefill):

449.5 tokens/s

quant:

Q3_K_L (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 40.9 t/s generation and 449.5 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q3_K_L and Metal, ngl 99, on one participating GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

51.9 tokens/s

prompt processing (prefill):

491.7 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 51.9 t/s generation and 491.7 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with IQ4_XS and Metal, ngl 99, using the author's locally converted weights. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

53.2 tokens/s

prompt processing (prefill):

520.6 tokens/s

quant:

IQ4_NL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 53.2 t/s generation and 520.6 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with IQ4_NL and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

61.4 tokens/s

prompt processing (prefill):

536.4 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 61.4 t/s generation and 536.4 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q4_0 and Metal, ngl 99, on one participating GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output chat request; TTFT was not measured.

generation:

52.0 tokens/s

prompt processing (prefill):

474.3 tokens/s

quant:

Q4_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 52 t/s generation and 474.3 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q4_K_S and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

49.2 tokens/s

prompt processing (prefill):

469.9 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 49.2 t/s generation and 469.9 t/s prompt processing on an M1 Max Mac Studio with 64GB unified memory. Setup is llama.cpp b2296 with Q4_K_M and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

57.2 tokens/s

prompt processing (prefill):

534.1 tokens/s

quant:

Q4_1 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 57.2 t/s generation and 534.1 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q4_1 and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one chat request; TTFT was not measured.

generation:

40.3 tokens/s

prompt processing (prefill):

422.9 tokens/s

quant:

Q5_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 40.3 t/s generation and 422.9 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q5_K_S and Metal, ngl 99, on one GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

39.7 tokens/s

prompt processing (prefill):

428.6 tokens/s

quant:

Q5_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 39.7 t/s generation and 428.6 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q5_K_M and Metal, ngl 99, on one GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

41.9 tokens/s

prompt processing (prefill):

455.7 tokens/s

quant:

Q5_1 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 41.9 t/s generation and 455.7 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q5_1 quantization and Metal backend at ngl 99, using one GPU. The figures come from separate llama-bench phases, PP512 and TG128, not a single chat request; TTFT was not measured.

generation:

39.0 tokens/s

prompt processing (prefill):

444.3 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 39 t/s generation and 444.3 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q6_K and Metal, ngl 99, on one GPU. The figures are separate llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

40.4 tokens/s

prompt processing (prefill):

547.3 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 40.4 t/s generation and 547.3 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with Q8_0 and Metal, ngl 99, on one participating GPU. The figures are separate synthetic llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

23.2 tokens/s

prompt processing (prefill):

603.1 tokens/s

quant:

F16 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 23.2 t/s generation and 603.1 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with F16 weights and Metal, ngl 99, on one participating GPU. The figures are separate llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

12.6 tokens/s

prompt processing (prefill):

549.3 tokens/s

quant:

F32 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat at 12.6 t/s generation and 549.3 t/s prompt processing on an M1 Max Mac Studio with 64 GB unified memory. Setup is llama.cpp b2296 with F32 weights and Metal backend at ngl 99, using one GPU. The figures come from separate llama-bench phases, PP512 and TG128, not one 512-input/128-output request; TTFT was not measured.

generation:

72.4 tokens/s

prompt processing (prefill):

37.8 tokens/s

quant:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Chat Q4_0 at 72.4 t/s generation and 37.8 t/s prompt processing on a 2022 Mac Studio M1 Ultra with 64 GB unified RAM. Setup is native llama.cpp build 944 with 512 context, batch 512, ngl 32, and 16 CPU threads; the source rates were 72.39 t/s from 27 evaluation runs and 37.75 t/s from 19 tokens. Single-token evaluation runs on Metal while multi-token prompt evaluation falls back to CPU, so the prompt figure is not GPU-prefill throughput.

generation:

370.7 tokens/s

prompt processing (prefill):

12605.1 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports SmolLM-135M-Instruct-4bit at 370.7 t/s generation and 12605.1 t/s prompt processing on an M1 Ultra 128GB. Setup is mlxcel 0.7.0-beta.1 with 4-bit weights and FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy, no speculative decoding. Prefill excludes loading, tokenization and client delivery, so it is not end-to-end TTFT; the decode rate covers 128 output tokens over 127 forward steps.

generation:

81.2 tokens/s

prompt processing (prefill):

518.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 8B 4-bit at 81.2 t/s generation and 518 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy temperature 0. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.