Mac M5 reports

generation:

34.5 tokens/s

quant:

6-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3-Coder-Next 80B-A3B at 34.5 tok/s decode on an M5 Pro 48GB Mac, running a full agentic coding task in 44s. Setup is the Global Agent engine (MLX-based) with 6-bit weights and expert streaming from SSD, speed mode, peak memory 31.5GB. The post also benchmarks Qwen3.8-Flash-Next 229B-A7B (19.2 tok/s), Qwen3-235B-A22B (9.19 tok/s) and MiniMax-M3 426B-A24B (4.62 tok/s) on the same machine, plus 16GB and 32GB memory-capped runs; the 16GB Qwen3-Coder-Next run reached 41.2 tok/s. Exact mode figures are lower (17.7, 7.88, 2.62, 2.20 tok/s).

generation:

61.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash (320B) at ~61 tok/s generation on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp with GLM's built-in MTP head (probabilistic drafting, 3 draft tokens); without MTP generation is 39.7 tok/s. The full 262K context fits with ~15 GiB to spare. Prompt reading is ~765 tok/s at 4K tokens and ~630 tok/s at 59K; a 29K-token prompt takes 45 s. Long-context recall was correct at 58.8K tokens. Images and MTP cannot be enabled simultaneously, and deep context on Metal needs an unmerged fix.

generation:

36.1 tokens/s

prompt processing (prefill):

498.3 tokens/s

quant:

2.44bit-mixed (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 36.1 t/s generation and 498.3 t/s prefill on a MacBook Pro M5 Max 128GB 40 GPU. Setup is oMLX 0.5.7 with a 2.44 bpw mixed MLX quantization (92.8 GB on disk) at 1k prompt, single request, 128 generated tokens, speculative decoding off. Generation holds 33.8 t/s at 4k and 31.5 t/s at 32k, with peak memory rising 1.03 GB across an 8x context jump; prefill at 32k takes 97 seconds to first token. Continuous batching reaches 83.2 t/s at batch 8 (2.30x).

generation:

205.0 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Gemma 4 E2B at 205 t/s decode at 4k context on an Apple M5 Max with 128 GB unified memory. Setup is mlx-vlm 0.4.4 with MLX 0.31.1, 4bit weights, and TurboQuant 2.5-bit KV cache quantization; the model uses 4.7 GB at 4k. Decode falls to 78 t/s at 256k context. TurboQuant gives +15-19% at 128-256k on larger models but is negative on E2B. The same report also covers Gemma 4 E4B, Gemma 4 26B-A4B, Gemma 4 31B, Qwen 3.5 2B and Qwen 3.5 4B.

generation:

20.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next-Heretic Q8_0 at 20.45 tok/s on an M5 Max 128GB. Setup is llama.cpp (closed PR #27739 fork) with Q8_0 and Q4_0 KV cache, -ngl 16, 4096 context, 45.1 GiB wired, MTP draft head enabled. Measured on 12 varied prompts with 24 warm-up queries discarded. The model is 122B total / 10B active MoE, 197 GB across 5 shards. A 262144-token server ran on llama.cpp v0.6.0 with ~32 tok/s prefill and decode falling to ~3.6 tok/s at 10k tokens. Under memory pressure a larger GPU tier wins: -ngl 24 at 10.29 tok/s with 15 GiB held by other work versus -ngl 6 at 5.75 tok/s with 45 GiB held. Contiguous layer placement beats scattered expert placement by 28.6%. MTP acceptance is 51-83%, helping +44% when experts are cached in RAM but costing 16-33% when they stream from SSD. Stock llama.cpp was 11-19% faster than the fork at 4k/64k/256k, and v0.6.0 another ~7% faster. No quality measurement was run for this file.

generation:

47.6 tokens/s

prompt processing (prefill):

1672.8 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 47.597 t/s generation and 1672.768 t/s prompt processing on an M5 Ultra 256GB Mac Studio. Setup is native mlx_lm.benchmark (mlx-lm 0.31.3 on MLX 0.32.2) with 4-bit MLX weights, 32768 prompt tokens and 128 generated tokens over 5 trials, peak memory 20.143 GB. Figures are medians from a third-party review unit; generation ranged 47.558-47.610 t/s and prompt processing 1671.556-1673.415 t/s. Speed only, no quality evaluation, and no hardware speedup ratio is claimed.

generation:

53.0 tokens/s

prompt processing (prefill):

2200.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 53 t/s generation and 2,200 t/s prefill on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is MTPLX 2.12.0 with speculative decoding and 55-65% draft acceptance; quantization and KV cache are unspecified. The 53 t/s figure is at roughly 4K context, with 51 t/s at roughly 16K context; the 2,200 t/s prefill is at a roughly 17K prompt. Resident memory was not measured; the 99 GB figure is the on-disk size only. Measurements are medians of three runs per cell, and the author notes small models served traffic on the same GPU throughout testing.

generation:

0.5-0.6 tokens/s

quant:

VQ3R

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi K3 2.78T running at 0.45-0.62 tok/s on a 64 GB MacBook Pro with M5 Pro, streaming experts from internal NVMe. Setup is the WARP engine with VQ3R 3-bit expert quantization, a 46.39 GB memory budget including a 17.56 GB expert cache, and a 982 GB converted container. The 64-token run is a range; the 200-token column is absent because one run takes ten minutes and reads 4.6 TB. The engine also runs DeepSeek-V4.1-Flash 552B at 3.77/3.71 tok/s and GLM-5.3-Flash 313B at 3.32/3.86 tok/s on the same machine.

generation:

70.0 tokens/s

quant:

8-bit

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 70+ tok/s generation on an M5 Max 128GB MacBook Pro. Setup is the MLTF engine with 8-bit weights and an INT8 KV cache, using Metal TensorOps W8A8 kernels. Cold prefill is 1,000+ tok/s on a 2Ki prompt, and aggregate generation throughput is 200+ tok/s at concurrency 4. The user chose 8-bit over 4-bit for quality.

generation:

86.7 tokens/s

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next Q2 at 86.7 tok/s with MTP on an M5 Max 128GB, using sf-q3-8flash, their stripped-down fork of antirez's ds4 engine. Setup is the sf-q3-8flash fork of ds4 on Metal with Q2 weights, MTP speculative decoding on, the whole model in unified memory. Stock ds4 ran the same Q2 MTP setup at 75.8 tok/s; on Q4 MTP went from 77.8 to 85.9 tok/s. No plain decode figure is given, only a 9-13% decode gain on Q2 and 2-11% prefill gain on Q4. Output stays bit-exact against stock ds4. An optional SSD streaming mode for experts on a simulated 48GB machine ran Q2 at 27 tok/s, around 35 tok/s with MTP.

prompt processing (prefill):

461.0 tokens/s

quant:

affine 4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-27B affine 4bit prefilling a 4096-token prompt at 461 tok/s on an M5 Pro 48GB (20-core GPU) with stock mlx 0.32.0 and mlx_lm 0.31.3 in a clean venv. The post is a bug report against oMLX 0.5.0, whose bundled MLX build fails a NAX metallib scan, so its M5 dispatch never engages: the same model prefilled at only 180 tok/s at 4096 tokens there, and 262 tok/s with OMLX_NAX=1. No decode figure is given beyond saying decode is normal. Qwen3.5-35B-A3B 4bit prefilled at 1,798 tok/s in the clean venv against about 900 on oMLX 0.5.0.

prompt processing (prefill):

1354.0 tokens/s

quant:

iQ-MLX 3.3 bpw (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on an M5 Ultra 96GB (64-core GPU) prefilling a cold 3.7k-token prompt at 1,354 tok/s. Setup is mlx-serve 26.10.1 with iQ-MLX 3.3 bpw, 131,072 context, native MTP adaptive decoding, PLD and a hot prefix cache, one request at a time. Decode is given per workload only: 227.5 tok/s on copy, 172.8 on new code and 106.4 on novel prose, with speculation on. Cold prefill rose to 4,083 tok/s at 25.2k. An agent replay ran at 125.8 tok/s text-round decode.

generation:

130.6 tokens/s

prompt processing (prefill):

5402.4 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 30B-A3B at 130.628 t/s generation and 5402.434 t/s prompt processing on an Apple M5 Max with 128GB unified memory. Setup is MLX 0.32.2 with MLX-LM 0.31.3 on macOS 27.0; quantization, context length and KV cache settings are not reported. Figures come from a third-party hardware review (Tbreak, September 21, 2026), not a reproducible benchmark log; prompt and generated token counts are unknown.

generation:

24.4 tokens/s

prompt processing (prefill):

636.0 tokens/s

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at Q2 running on an M5 Max 128GB Mac, with a fork of ds4 reaching 24.4 t/s generation at 2048 context versus 12.1 t/s for stock ds4. Setup is ds4 with a 341 GB GGUF (152 GB weights plus 189 GB Engram tables) streamed from SSD, no KV quant, bit-exact output. The fork doubles steady decode to 25.7 t/s at 2048 context and 22.0 t/s at 32768 context, and prefill from 16k to 32k context rises from 404 t/s to 636 t/s. Adding a byte-identical external Thunderbolt 5 SSD copy cuts time to first token on a 3.5K prompt from 13.2 s to 11.2 s and after an 8K context from 1.38 s to 0.18 s; decode is unchanged. The enclosure runs at PCIe 4.0 x4, about half the internal SSD speed. A two-SSD attempt in upstream ds4's mmap path was bit-exact but 13-38% slower on prefill and was not submitted.

generation:

39.9 tokens/s

quant:

8-bit (MLX)

kv:

4-bit / 8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 39.9 tok/s with DFlash 2 speculative decoding on an M5 Max 128GB, versus 17.9 tok/s plain decode. Setup is mlx-dspark 0.14 on MLX 0.32 with the 8-bit MLX build (~29 GB) and 4-bit/8-bit quantized KV cache; the DFlash 2 drafter is incoai/Qwen3.8-27B-DFlash2 (3.8 GB). In the agent harness the model runs 20-29 tok/s. A bf16 run of the same model measured 9.7 tok/s plain and 36.5 tok/s with DFlash 2 (4.43 accepted tokens per round, 137 target forwards, output identical). Agent-12 scores with DFlash 2 were 12/12 easy in 122 s and 7/8 hard in 401 s, about 3x faster than the Aug 18 run at the same accuracy. The Claude Code smoke test passed in 38 s. Gemma 4 31B Abliterated 4-bit ran at 26 tok/s and Gemma 4 E4B 4-bit failed the smoke test. On an M4 Pro 64 GB, Gemma 4 31B runs at ~13.5 tok/s and server-side prompt fixes cut hello latency from ~120 s to ~3-5 s.

generation:

1.0 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi K3 (2.78T) at 0.96 tok/s on a MacBook Pro M5 Max 128GB, with the trunk in memory and 1.44 TB of 4-bit routed experts streamed from NVMe drives. Setup is a llama.cpp-derived fork with experts split across the internal SSD plus up to three Thunderbolt 5 enclosures, each holding a byte-identical replica. Decode scales with drive count: 0.55 tok/s on one drive, 0.75 with one enclosure, 0.89 with two, 0.96 with three (1.75x), and 1.00 steady over 512 generated tokens, 1.13 over 128.

generation:

58.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 58 tok/s generation and 2,000 tok/s prefill on a Mac Studio M5 Ultra 256GB. Setup is MTPLX 2.12.0 with speculative decoding, 55 to 65 percent draft acceptance, 99 GB resident, 4k context. Four models run in slots on the same box; Gemma 4 26B 4-bit reaches 113 tok/s, Qwen 3.6 35B-A3B 114 tok/s, DeepSeek V4 Flash 4-bit 31 tok/s. DeepSeek prefill is from a single uncached 13,139-token request in 21.8 s. Clef-Flash 9B decision slot has a 259 ms median.

generation:

76.7 tokens/s

prompt processing (prefill):

792.4 tokens/s

quant:

MXFP4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B AXQ MXFP4 MTP at 76.70 tok/s decode and 792.4 tok/s prefill on an M5 Max 128 GB. Setup is AX Engine 7.5.3 with MXFP4 MTP weights, median of 20 runs on the flappy suite. The same campaign measured the 6-bit pack at 76.04 / 768.7 and long_code at 72.71 / 856.9; MTPLX 2.11.2 reached 63.91 / 697.3 and mlx-lm 34.20 on the MXFP4 pack.

generation:

35.3 tokens/s

prompt processing (prefill):

528.9 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B at 35.28 t/s decode and 528.91 t/s prefill on an M5 Pro 64GB Mac Mini. Setup is Splash with Q8_0 GGUF weights and 4k input prompts. User also compares llama.cpp, MLX, and Splash across several quants, and reports an M1 Max 64GB Splash run at 135.96 t/s decode and 915.01 t/s prefill. User settled on Splash with Q8_0 and is happy with the results.

generation:

73.5 tokens/s

prompt processing (prefill):

1453.0 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 73.5 tok/s with MTP speculative decoding on an M5 Max 128GB, versus 43.8 tok/s plain autoregressive. Setup is MTPLX with a dynamic 4-bit quant (8-bit attention) MLX pack, 262,144-token context, MTP depth ceiling 3, single stream through mtplx serve. Prompt processing on a 4,061-token prompt is 1,453 tok/s on MTPLX 2.12.0 (786 tok/s on 2.11.3); a 65,502-token prompt reads at 1,094 tok/s with 63.5 tok/s decode. The pack downloads at 115.1 GB including a 32 GB n-gram table streamed from SSD, peaks at 87 GiB while serving, and is recommended for Macs with 128 GB or more.