Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target plus DFlash 2 draft) on a MacBook Pro M5 Max with 48 GB, using a Python code context and 128 generated tokens per test.
At a 4096-token prompt the single request reaches 93.8 t/s decode and 1207.0 t/s prefill; longer prompts give 189.4 t/s decode at 16382 tokens and 110.2 t/s at 65536 tokens.
Batched continuous batching reaches 413.7 t/s aggregate at 2x and 407.0 t/s at 4x, with peak memory between 19.1 GB and 22.2 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports running Qwen3.8 27B (4-bit, ~16GB) in LM Studio on a MacBook Pro M5 Max with 48GB unified memory, at around 10-15 tokens/sec.
Setup is LM Studio serving an OpenAI-compatible API to the Pi Coding Agent, which built a React + TypeScript dashboard from scratch over a few hours.
The user says local coding agents are capable but not a drop-in replacement for Claude Code or Codex with frontier models, citing slower inference and reliability concerns.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports QwQ-32B at 26.3 t/s generation and 685.4 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention enabled, and n_gpu_layers=99.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 124.3 t/s generation and 3342.7 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 9B at 86.5 t/s generation and 2369.3 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 4B at 135.6 t/s generation and 4099.7 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 111.2 t/s generation and 3378.3 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-27B at 29.2 t/s generation and 730.5 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention enabled and n_gpu_layers=99; the context length is unspecified.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 8B at 97.7 t/s generation and 2744.7 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 4B at 159.7 t/s generation and 4972.7 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 32B at 26.4 t/s generation and 667 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99.
The figures are separate llama-bench microtests, tg128=26.38 t/s and pp512=667.0 t/s, not one 512-input/128-output request; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 30B A3B at 148.5 t/s generation and 3496.9 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 14B at 57.4 t/s generation and 1584.3 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 1.7B at 288.6 t/s generation and 11493.01 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3 0.6B at 457.1 t/s generation and 23288.5 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 7B at 106 t/s generation on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 32B at 26.3 t/s generation and 685 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention on, and n_gpu_layers=99.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 2.5 Coder 14B at 56.0 t/s generation and 1545.95 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory.
Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Phi-4 at 56.8 t/s generation and 1537.5 t/s prompt processing on an M5 Max with 48GB unified memory.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Phi-4 at 164.9 t/s generation and 5412.7 t/s prompt processing on an M5 Max 48GB.
Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true.
The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.