Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory.
Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing.
The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Ornith 1.5 35B-A3B at 108 tok/s on an M2 Max 32GB, versus 66 tok/s with the macOS Aerial lock-screen animation playing.
Setup is oMLX with a 4-bit MoE model, medians of 4-6 local runs over 512 output tokens.
After an 8k-token prompt, generation was 89 vs 66 tok/s, and WindowServer CPU use rose from roughly 25% to 44% while the animation ran. Using caffeinate -i instead of -d let the display sleep and restored speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 83.75 t/s on an Apple M2 Max 64GB, the middle of three sequential runs (82.28, 83.75, 83.09 t/s).
Setup is LM Studio with the MLX runtime (mlx-llm-mac-arm64-apple-metal-advsimd 1.13.1), 4-bit MLX weights, 262144-token context, parallel 4, whole model in unified memory.
One request at a time with no speculative decoding; 705 prompt tokens and 255 completion tokens, all reasoning tokens. Run 1 was cold with 6.973 s to first token, warm runs 1.156 s and 0.897 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B at 17.2 t/s generation and 63.7 t/s prompt processing on a Mac Studio M2 Max with 64 GB unified memory.
Setup is mlx-vlm 0.4.4 with 4-bit affine group-64 weights and MLX 0.31.1 on one GPU, running the text-only long-text request with 55 prompt tokens and 500 generated tokens.
The 29.9-second whole generate call is not TTFT, and the 63.7 t/s prompt figure is a single request rather than a llama-bench pairing.
LLaMA family 7B / Q4_0 / Mac mini M2 Metal b1550 / PP512 TG128 / exact base and artifact unknown
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a LLaMA-family 7B model at 21.9 t/s generation and 179.6 t/s prompt processing on a Mac mini M2 with 24 GB unified RAM.
Setup is llama.cpp build 1550 with Q4_0 quantization, F16 KV cache, Metal backend and ngl 99, running llama-bench PP512 and TG128.
The figures are independent synthetic prompt and generation tests, not a 512-token prompt followed by a 128-token answer; the source rates are 179.57 ± 0.04 t/s and 21.91 ± 0.02 t/s.
LLaMA family 7B / Q8_0 / Mac mini M2 Metal b1550 / PP512 TG128 / exact base and artifact unknown
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a LLaMA-family 7B model mostly Q8_0 at 12.2 t/s generation and 181.4 t/s prompt processing on a Mac mini M2 with 24 GB unified RAM.
Setup is llama.cpp build 1550 with Q8_0 and F16 KV cache, Metal backend at ngl 99, one GPU.
The figures are native llama-bench PP512 and TG128 synthetic tests, not a 512-token prompt followed by a 128-token answer; the source rates are 181.40 ± 0.05 t/s and 12.21 ± 0.04 t/s.
LLaMA family 7B / F16 / Mac mini M2 Metal b1550 / PP512 TG128 / exact base and artifact unknown
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a LLaMA-family 7B model at 6.7 t/s generation and 201.3 t/s prompt processing on a Mac mini M2 with 24 GB unified RAM.
Setup is llama.cpp build 1550 with F16 weights and F16 KV cache, Metal backend with ngl 99 on one GPU.
The figures are native llama-bench PP512 and TG128 results, 201.34 ± 0.21 t/s and 6.72 ± 0.01 t/s, independent synthetic prompt and generation tests rather than a 512-token prompt followed by a 128-token answer.
CodeLlama 70B base / Q5_K_M / M2 Ultra Metal b2050 / native 9-token prompt and 399 eval runs
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports CodeLlama-70B at 8.7 t/s generation and 15.1 t/s prompt processing on a Mac Studio M2 Ultra with 64 GB unified memory.
Setup is llama.cpp build 2050 with Q5_K_M and F16 K/V cache, context and batch 512, 16 CPU threads, and 81/81 layers offloaded to the single Metal device.
The source prints 8.72 t/s generation over 399 evaluation runs and 15.06 t/s prompt processing over 9 tokens; the stored figures are rounded to 0.1 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Code Llama 7B Instruct at 33.6 t/s generation and 200.7 t/s prompt processing on an M2 Pro 32GB Mac mini.
Setup is llama.cpp with Metal, Q4_K_M GGUF, one Metal device, and a Japanese Fibonacci-code prompt at --temp 0.1.
The figures are native eval phase timers from a single displayed run, not an end-to-end average; context length, layer count, and KV precision are unreported.
ALMA 7B-Ja / Q4_K_M / M2 Pro 19-core / native phases
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports ALMA-7B-Ja at 34.8 t/s generation and 227.6 t/s prompt processing on an M2 Pro 32GB.
Setup is llama.cpp with Metal, Q4_K_M GGUF, run with -n 128; layer count, context length, batch size and KV precision are unreported.
The figures are native request phase timers from one displayed run, not an end-to-end average, with no warmup or cache protocol.
M2 Pro 16GB · llama-cpp-python 0.2.29 (llama.cpp Metal backend) · 10,240 ctx
generation:
12.0 tokens/s
prompt processing (prefill):
8.7 tokens/s
quant:
Q5_K_M (GGUF)
kv:
F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-7B-OpenOrca at 12 t/s generation and 8.7 t/s prompt eval on an M2 Pro with 16GB unified RAM.
Setup is llama-cpp-python 0.2.29 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048, with only 1 of 33 layers offloaded to Metal.
The source's exact rates are 11.96 t/s over 207 eval runs and 8.72 t/s for 35 prompt tokens; the run is a single trial with partial offload, not a full-GPU benchmark.
M2 Pro 16GB · llama-cpp-python 0.2.28 (llama.cpp Metal backend) · 10,240 ctx
generation:
25.3 tokens/s
prompt processing (prefill):
13.0 tokens/s
quant:
Q5_K_M (GGUF)
kv:
F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-7B-OpenOrca at 25.3 t/s generation and 13 t/s prompt processing on an M2 Pro with 16GB.
Setup is llama-cpp-python 0.2.28 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048.
The 25.31 t/s comes from 207 eval runs over 8177.10 ms, and the 12.98 t/s prompt rate from 35 tokens in 2696.68 ms; n_gpu_layers=1, so full-layer offload is not asserted.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-Coder 30B-A3B at 71.2 t/s generation and 1096.8 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory.
Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention enabled, run with -ngl 100 on one 60-core GPU.
The figures are separate synthetic llama-bench tests: PP512 at 1096.84 +/- 5.48 t/s and TG128 at 71.23 +/- 0.08 t/s, with 16 CPU threads and a 16.45 GiB model footprint.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-Coder 30B-A3B at 69.3 t/s generation and 1080.5 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory.
Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention off, one GPU with -ngl 100 and 16 threads.
The figures are separate synthetic llama-bench tests: PP512 and TG128, not a 512-input/128-output request. The same run measured 818.85 t/s PP8192 and 40.35 t/s TG8192.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Cyber-Tiel-Coder-35B-A3B at 152.5 t/s on an M2 Ultra 192GB Mac Studio.
Setup is the AX Engine with an AXQ MXFP4 trunk and protected MTP sidecar, one Metal GPU, expert streaming off.
The figure is the median decode rate over six samples on a 219-input/256-output python-lru workload, with TTFT 505.93 ms; completion including TTFT was 117.59 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Nemotron 3 at 93 t/s generation and 468.6 t/s prompt processing on an Apple M2 Max with 64GB unified memory.
Setup is MLX with 4-bit quantization, a requested prompt length of 512 and generation cap of 128 at batch size 1, with no HTTP server involved.
The figures are the mean of three retained trials after discarding Trial 1; peak memory was 18.974 in engine output units, and KV cache precision and prefill chunk size are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 32B at 18 t/s generation and 126.8 t/s prompt processing on an M2 Max 64GB.
Setup is MLX with 4-bit quantization, batch size 1, a requested 512-token prompt and 128-token generation cap, with peak memory 19.12 in engine output units.
The stored figures are the three-trial means (TG 17.96 t/s, PP 126.82 t/s) rounded to 0.1 t/s; the source's CLI average includes a discarded fourth trial.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 35B-A3B at 88.9 t/s generation and 521.5 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory.
Setup is MLX with 4-bit quantization, a requested 512-token prompt and 128-token generation cap at batch size 1, with no drafter passed.
The figures are three-trial means after discarding Trial 1; the source means are 88.85 t/s generation and 521.47 t/s prompt, and peak memory was 20.218 in engine output units.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 27B at 20.7 t/s generation and 91.4 t/s prompt processing on an Apple M2 Max with 64GB unified memory.
Setup is MLX with 4-bit quantization, batch size 1, a requested prompt length of 512 and generation cap of 128, run directly via python -m mlx_lm benchmark with no HTTP server.
The figures are the mean of three retained trials after discarding Trial 1; the source reports 20.71 t/s TG and 91.36 t/s PP, and peak memory of 16.302 in engine output units.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-4.7 at 52.6 t/s generation and 413.7 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory.
Setup is MLX with a 4-bit artifact, batch size 1, requested 512-token prompt and 128-token generation; KV cache precision and prefill chunk size are unreported.
The figures are three-trial means after discarding Trial 1, with sample SD 0.04 for TG and 0.38 for PP; peak memory was 17.497 in engine output units.