Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports Qwen3-14B writing at 18 tok/s on a MacBook Pro M2 Pro with 16GB, where the model fits entirely in unified memory.
Setup is llama.cpp with the Metal build, run through the MAFDET app, which reasons about the message while the user is still typing.
The app cuts the wait from Enter to the first word from 32.7 s to 12.1 s on 6 word problems, with 6/6 accuracy both ways; on Qwen3-4B the mean/median wait drops from 18.1s/16.7s to 5.6s/4.1s over 24 problems. The same 14B on the user's RTX 2060 is split with CPU at 4.5-5 tok/s. Tested on one Mac only.
M2 Pro 16GB · llama-cpp-python 0.2.29 (llama.cpp Metal backend) · 10,240 ctx
generation:
12.0 tokens/s
prompt processing (prefill):
8.7 tokens/s
quant:
Q5_K_M (GGUF)
kv:
F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-7B-OpenOrca at 12 t/s generation and 8.7 t/s prompt eval on an M2 Pro with 16GB unified RAM.
Setup is llama-cpp-python 0.2.29 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048, with only 1 of 33 layers offloaded to Metal.
The source's exact rates are 11.96 t/s over 207 eval runs and 8.72 t/s for 35 prompt tokens; the run is a single trial with partial offload, not a full-GPU benchmark.
M2 Pro 16GB · llama-cpp-python 0.2.28 (llama.cpp Metal backend) · 10,240 ctx
generation:
25.3 tokens/s
prompt processing (prefill):
13.0 tokens/s
quant:
Q5_K_M (GGUF)
kv:
F16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Mistral-7B-OpenOrca at 25.3 t/s generation and 13 t/s prompt processing on an M2 Pro with 16GB.
Setup is llama-cpp-python 0.2.28 via LangChain with Q5_K_M, F16 K/V cache of 1280 MiB, 10240-token context and batch 2048.
The 25.31 t/s comes from 207 eval runs over 8177.10 ms, and the 12.98 t/s prompt rate from 35 tokens in 2696.68 ms; n_gpu_layers=1, so full-layer offload is not asserted.