Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 61.6 t/s generation and 465.2 t/s prompt processing on an Apple M2 Max with 64GB of shared unified memory.
Setup is MLX via the mlx_vlm text-only helper with a 4-bit mixed config, batch_size=1, requested 512-token prompt and 128-token generation cap, no HTTP server.
The figures are three-trial means after discarding Trial 1; retained generation values were 61.389, 61.732 and 61.742 t/s, and peak memory was 15.255 GiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports LFM2 24B-A2B at 117.8 t/s generation and 616.4 t/s prompt processing on an M2 Max 64GB.
Setup is MLX with a 4-bit mixed config, run directly via python -m mlx_lm benchmark with batch_size=1, a 512-token prompt and 128-token generation cap.
The figures are three-trial means after discarding Trial 1; peak memory was 13.822 in engine output units, and KV cache precision, prefill chunk size and sampling settings are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Devstral-Small-2505-4bit at 24.4 t/s generation and 104.2 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory.
Setup is MLX via python -m mlx_lm benchmark with 4-bit group 64 weights, a requested 512-token prompt and 128-token generation cap at batch size 1; KV cache precision is unreported.
The figures are three-trial means after discarding Trial 1, with retained TG values of 24.434, 24.417 and 24.424 t/s; peak memory was 13.763 in engine units.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports gpt-oss 21B-A3.6B at 85.5 t/s generation and 501.5 t/s prompt processing on an M2 Max with 64GB unified memory.
Setup is MLX with MXFP4-Q4 mixed quantization, run directly via python -m mlx_lm benchmark with batch_size=1, a requested prompt length of 512 and generation cap of 128.
The figures are three-trial means after discarding Trial 1; peak memory was 11.713 in engine output units, and KV cache precision, prefill chunk size and sampling settings are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 9B at 62.7 t/s generation and 321.2 t/s prompt processing on an Apple M2 Max with 64GB unified memory.
Setup is MLX with 4-bit quantization, a requested prompt length of 512 and generation cap of 128, batch size 1, no HTTP server.
The figures are three-trial means after discarding Trial 1; retained generation trials were 62.651, 62.747 and 62.627 t/s, and peak memory was 5.87 in engine output units.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Llama 3.1 8B at 72.9 t/s generation and 566.4 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory.
Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, batch size 1, with a requested 512-token prompt and 128-token generation cap.
The figures are three-trial means after discarding Trial 1; peak memory was 5.116 in engine output units, and KV cache precision and prefill chunk size are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Granite 3.2 at 64.4 t/s generation and 491.1 t/s prompt processing on an M2 Max 64GB.
Setup is MLX with 4-bit quantization, batch size 1, a requested 512-token prompt and 128-token generation cap, with no HTTP server involved.
The figures are three-trial means after discarding Trial 1; peak memory was 5.218 in engine output units, and KV cache precision and prefill chunk size are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen2.5 7B at 76.8 t/s generation and 602.8 t/s prompt processing on an Apple M2 Max with 64GB unified memory.
Setup is MLX via python -m mlx_lm benchmark with the mlx-community/Qwen2.5-7B-Instruct-4bit artifact, batch size 1, 512-token prompt and 128-token generation cap.
The 76.8 t/s and 602.8 t/s are three-trial means after discarding Trial 1; retained generation values were 76.773, 76.747 and 76.821 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Ministral 3 3B at 128.3 t/s generation and 743.4 t/s prompt processing on an Apple M2 Max with 64 GB unified memory.
Setup is MLX via a direct python -m mlx_lm benchmark, 4-bit quantized, with a requested prompt length of 512 and generation cap of 128 at batch_size=1.
The figures are three-trial means after discarding Trial 1; peak memory was 2.523 in engine output units, and KV cache precision and prefill chunk size are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 61.7 t/s generation and 276.8 t/s prompt processing on a Mac mini with Apple M2 Pro and 32 GB unified memory.
Setup is mlx-lm 0.31.3 with MLX 0.31.2, MLX 4-bit group-64 weights of about 19.0 GiB, temperature 0 and --max-tokens 128.
The figures are medians of three runs on a 490-token prompt with 128 tokens generated; peak memory was 20.261 GB per run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 26.4 t/s generation and 276.2 t/s prefill on an Apple Mac Studio M2 Ultra with 128 GB unified memory.
Setup is oMLX 0.6.3 with oQ8e mixed affine 8-bit plus FP16 weights, run as an ANE and CPU split at a source-labelled 65,536-token prompt depth.
The GPU-only control at the same depth is 231.2 prefill and 26.6 decode; the author reports gains apply to cold prefills, not prefix-cached turns, and peak memory at 64K rises from 44.0 GB to 60.0 GB with ANE.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Tiel-Coder 35B-A3B at 134.6 t/s median native decode on a single Apple Mac Studio M2 Ultra with 192 GiB unified memory.
Setup is AX Engine with the AXQ-MXFP4 MLX export, explicit throughput-MTP at maximum draft depth three, expert streaming off, and the model kept fully resident.
Separate median completion throughput is 107.28 t/s including TTFT, with median native TTFT 494.41 ms; the 256-token output is total output, not the decode numerator.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 22.2 t/s generation and 325.3 t/s prefill on an M2 Ultra 192GB.
Setup is AX Engine at MXFP4 / affine 8-bit mixed, speculative decoding on, one request.
Medians of 20 trials in defai-digital's AutomatosX benchmark of 2026-10-08. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports MiniMax M2.7 at 32.7 t/s generation and 420.8 t/s prefill on an M2 Ultra 192GB.
Setup is llama.cpp at UD-Q4_K_M, one request, speculative decoding off.
Token-weighted rates across four sequential agentic tasks in crypdick's run of 2026-08-10. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports MiMo-V2.5 at 37.87 t/s generation and 301.61 t/s prompt processing on an M2 Ultra 192GB Mac Studio.
Setup is llama.cpp llama-server with UD-Q3_K_M GGUF, q8_0 KV cache, 65,536 context, Flash Attention on, gpu_layers 999, batch 2048, and 8,192 MiB cache_ram.
Figures are token/time-weighted across four sequential real-world agentic tasks (n=1 each), not llama-bench or end-to-end throughput; one task hit its 8,192-token output cap and 258 of 2,290 prompt tokens were cached. Server peak RSS was 131.015 GiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
summarizationagenticlong-context
User reports Qwen3.8 27B at 23 t/s generation and 304.2 t/s prompt processing on an M2 Ultra 128GB Mac Studio.
Setup is oMLX 0.6.3 with oQ4e 4-bit quantization at workload depth 65536, with 16.7GiB of weights and peak memory rising from 25.3 to 36.2GB.
The 304.2 PP is the ANE-on arm; the separate GPU-only baseline is 242.2 PP and 23.8 TG, and the source calls the decode change within variance. Depth 65536 is not a verified configured maximum context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-use
User reports Qwen3.6-35B-A3B at 71.3 t/s decode on short prompts on a Mac Studio M2 Max 64GB.
Setup is Ollama with NVFP4 MLX weights, q8_0 KV cache, 32768 context, flash attention on, single concurrent request, think disabled.
Decode stays near-flat with prompt length (69.3 t/s medium, 67.8 t/s long) and the 5-turn multiturn total is 23.7 s; the model is the recommended default coding model, replacing Qwen3-Coder-30B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 58.56 t/s generation and 671.40 t/s prefill on an M2 Max 32GB.
Setup is llama.cpp build c1d0e7a00 with Metal backend, Q4_K_M GGUF (20.22 GiB), 4096-token prompt, no speculative decoding.
The dataset reports mostly negative results: MoE beats dense Qwen3.8-27B by 5.4x generation, aggressive quantization bought little, MTP and DFlash2 speculative decoding were net losses on Metal, and q8_0 KV cache was free. MLX reached 86.9 t/s generation but only 296 t/s prefill versus llama.cpp's 57.8 t/s and 685 t/s. oMLX with paged SSD KV cache reached 13.7 t/s generation and 84 t/s prefill on the dense 27B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares GGUF and MLX inference speeds on a Mac Studio M2 Ultra with 192GB, using LM Studio 0.3.10 with MLX 0.6.
For Qwen2.5 72B 8-bit, GGUF reaches 7.68 tokens/sec while MLX drops to 0.42 tokens/sec. Smaller models show the opposite: Qwen2.5 0.5B 4-bit runs at 79 tokens/sec in GGUF versus 317 tokens/sec in MLX, and Mistral Small 22B 8-bit is roughly equal at 25.01 versus 25.96 tokens/sec.
User reports MLX is 4-5x faster for very small models but about 10x slower for large ones, with the inflection point around 22B parameters, and notes MLX does not use much GPU for larger models while LM Studio lacks a layer-offload option for MLX.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks llama.cpp in a GPU-accelerated Podman container on an M2 Max Mac Studio with 64GB RAM, reporting 428.47 t/s prompt processing and 35.88 t/s token generation for llama 7B Q4_0.
Setup uses llama.cpp built with Vulkan and GPU-remoting via krunkit, with -ngl 99, in a Fedora 40 container.
The container is about 40% slower than native Metal (746.06 t/s pp512, 71.36 t/s tg128), but 3x faster than a CPU-only container in prompt processing and 25% faster in token generation. Enabling -fa in the Vulkan container drops performance to 312.92 t/s pp512 and 14.10 t/s tg128.