Mac M3 reports

generation:

70-100 tokens/s

prompt processing (prefill):

900-1000 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Flash-Next running through oMLX plus Splash on a Mac Studio M3 Ultra 96GB, used mainly for agentic coding with Pi. Generation is given as 70–100 tokens/s and prefill as 900–1,000 tokens/s, both sustained up to 200K tokens of context with no noticeable degradation. No quant, KV cache format or single measured figure is stated.

generation:

122.7 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B at 122.7 tok/s on an M3 Max 36GB with Kapitsa 0.2.12, tying oMLX at 122.9 tok/s. Setup is Kapitsa with a 4-bit MLX build and MTP head plus DFlash2 drafter, sustained 256-token prose, median of 3 runs. Other engines on the same model: Rapid-MLX 110.3, Splash 97.0, MTPLX 79.9 tok/s. On dense Qwen3.8-27B-4bit Kapitsa leads at 41.3 tok/s, ahead of oMLX 34.1, Splash 33.0, MTPLX 31.6 and Rapid 26.2. Flint 40B-A2.5B reaches 134.2 tok/s on Kapitsa and 103.0 on Rapid. Draft acceptance was 75-78% dense and 81-85% MoE. No formal quality eval was run.

generation:

4.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Spark Open 816B-A42B decoding at 4 t/s on an M3 Ultra 256GB. Setup is ds4; no quant, context length or KV cache is stated. The user calls this the first successful run that reached decode and hopes for a 4-5x improvement with larger prefill chunks.

generation:

77.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 77.8 t/s generation. Source evidence (verbatim): Source: https://terminalbytes.com/ollama-vs-llama-cpp-vs-mlx-mac-2026/ # Ollama vs llama.cpp vs MLX on a Mac Studio, measured Hardware: Mac Studio with an M3 Ultra and 256GB unified memory. The author ran Qwen3.5 9B through every runtime with the same 34-token prompt, thinking off, one request at a time, two runs each, no speculative decoding. Results, generation speed: llama.cpp llama-bench build 10809 with unsloth Qwen3.5-9B Q4_K_M GGUF, tg128 77.8 tok/s; Ollama 0.32.13 GGUF qwen3.5:9b Q4_K_M 77.7 tok/s; Ollama 0.32.13 MLX NVFP4 90.5 tok/s; rapid-mlx 0.14.1 Qwen3.5-9B 4-bit 105.9 tok/s; mlx-lm 0.31.3 Qwen3.5-9B 4-bit 107.5 tok/s. The author wrote down the numbers from his own runs.

generation:

73.4 tokens/s

prompt processing (prefill):

1808.0 tokens/s

quant:

Q8_0 (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-35B-A3B at 73.43 tokens/s generation and 1808 tokens/s prefill on an Apple M3 Ultra 96GB. Setup is llama.cpp build 8860 with Q8_0 GGUF weights and f16 KV cache, Metal backend, single machine. The figures are separate synthetic tests (TG128 and PP512), not an end-to-end request. Historical TG128 85.48, turbo3 KV 63.23 and PP8192 1183 belong to other configurations and are kept for comparison only.

generation:

51-59 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next decoding at a median of 51-59 tokens/s (p90 ~65) on two M3 Ultra 256GB Mac Studios. Setup is MTPLX with a 4-bit mixture-of-experts model (107 GB on disk) and multi-token prediction, carrying 116-142k-token prompts with a 97-99% prompt-cache hit rate. The user also compares DeepSeek V4.1 Flash against MiMo V2.6 Flash head to head (8 agents each, 3 hours): 11 vs 7 PRs merged and 1 vs 4 stalls. DeepSeek costs about 1.6x per output token but less per merged PR, so every builder now runs DeepSeek.

generation:

29.2 tokens/s

quant:

6-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 29.2 t/s generation. Source evidence (verbatim): Source: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and # First-hand Mac Studio M3 Ultra 512GB Qwen3.5-397B-A17B 6-bit mlx-vlm generation measurement Original first-hand report by AlooftWaffle, published March 27, 2026: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and . Historical source-reported measurements, not a new benchmark or independent reproduction. Selected configuration: one Mac Studio M3 Ultra with 512GB unified memory, running Qwen3.5-397B-A17B at publisher-described 6-bit precision using mlx-vlm. The shortest reported context row gives generation 29.2 tokens/s. The source labels this row 4K context; whether that label denotes actual occupied input or configured capacity is not established. Actual fresh input count, cached prefix, output count and runtime version are unknown. Leave structured context length unknown. Do not replace it with 4096 or infer zero cache. CPU/GPU core counts, sampling, thinking mode, speculative decoding, request concurrency, warmup, repeats, statistical aggregation, exact measurement date, OS, KV precision, Flash Attention, offload, temperatures, power and peak resident memory are unreported. The source's longer Mac rows report generation 28.6 tokens/s at the 8K label and 26.7 tokens/s at 32K; preserve these as separate source rows, not an average. The author also calls values 317/537/476 tokens/s 'prefill throughput (time to first token)'; native timing boundaries and counters are absent, so leave structured prompt throughput and TTFT unknown rather than converting or equating these metrics. Verified upstream model-family reference: https://huggingface.co/Qwen/Qwen3.5-397B-A17B . This is a base/reference repository, not the tested 6-bit artifact. The tested quantization repository, revision, file names, hashes and quantization recipe are unknown; do not assign a community pack or GGUF Q6 code. The source describes nearly 300GB of downloaded model files; that is download/storage volume, not measured resident RAM or a peak-memory measurement. The same article's dual-Spark INT4-AutoRound/vLLM measurements and embedding sentences/s are different hardware, quantization and workloads; none is the selected Mac generation speed or a controlled comparison isolating hardware. Full original source remains the URL above; no author contact or comments were made.

generation:

73.0 tokens/s

prompt processing (prefill):

347.0 tokens/s

quant:

MXFP4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 73 t/s generation and 347 t/s prompt processing on a Mac Studio M3 Ultra with 96GB, running mlx_lm.server with no speculative decoding and one request at a time. Source evidence (verbatim): Source: https://elitetestengineering.substack.com/p/gpt-oss-120b-on-two-generations-of # GPT-OSS-120B on Two Generations of Mac Studio: The Long-Context Gap Gets Bigger, Not Smaller Two Mac Studios running gpt-oss-120b (mlx-community/gpt-oss-120b-mxfp4-bf16, about 65GB) through mlx_lm.server. M5 Ultra: 30 CPU cores, 64-core GPU, 256GB unified memory. M3 Ultra: 28 CPU cores, 60-core GPU, 96GB unified memory. Short prompt of about 146 tokens: M5 Ultra prefill about 555 tok/s and decode 99 tok/s, M3 Ultra prefill about 347 tok/s and decode 73 tok/s. Long prompt of about 24,400 tokens: M5 Ultra prefill about 3,955 tok/s and decode 76 tok/s, M3 Ultra prefill about 1,101 tok/s and decode 55 tok/s. No speculative decoding was used. The M5 Ultra ran with decode concurrency set above 1 while the M3 Ultra ran with concurrency 1 and prompt caching off, and the article states the M5 Ultra setting inconsistently (2 and 4).

generation:

49.1 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 49.1 t/s generation. Source evidence (verbatim): Source: https://arxiv.org/html/2605.00519v1 # First-hand M3 Ultra 96GB Qwen3-Next 80B native MLX 4-bit generation measurement Original experimental paper Silicon Showdown, arXiv 2605.00519v1, May 2026. Selected Table 5 row: Apple M3 Ultra 96GB, 28 CPU cores, 60 GPU cores; installed unified memory 96GB. macOS 26 Tahoe, MLX v0.30.6. Qwen3-Next-80B-A3B native MLX 4-bit: generation 49.1 tokens/s, 0% CPU offload. Long prompt approximately 496 tokens; output 256 tokens, greedy decoding. Authors report arithmetic mean of three consecutive timed runs after two untimed warmups and fixed random seed. This is their published generation-phase measurement, not a speed derived from TTFT. Prompt-processing speed unreported; leave promptTps null. Context capacity, KV quantization, speculation, concurrency and resident allocation unreported. Verified base-model reference explicitly cited by the paper: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct . This is not the tested MLX quantization repository; exact quantized artifact and revision unknown. Do not infer disk size or resident memory. Table 2 names the 96GB machine, not the followed 256GB configuration. Other models, machines and Nvidia offload comparisons are separate runs. Source: https://arxiv.org/html/2605.00519v1 .

generation:

81.2 tokens/s

prompt processing (prefill):

2324.6 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B Q8_0 at 81.2 t/s generation and 2324.6 t/s prompt processing on a Mac Studio with an Apple M3 Ultra and 512 GiB unified memory. Setup is llama.cpp with Metal GPU offload, Flash Attention on, F16 KV cache, 8 threads, and batch/ubatch 512/512; the Mac-only control arm uses one GPU. PP4096 and TG128 are separate synthetic tests, with TG128 starting from an empty context rather than generation after prefill. The article also reports a continuous 4096-input plus 128-output workload at 3.409 seconds, and a separate tuning pass at the same batch/ubatch gives PP4096 2323.37 and TG128 80.68 t/s.

generation:

80.9 tokens/s

prompt processing (prefill):

2779.0 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-35B-A3B at 80.9 t/s generation and 2779 t/s prompt processing on a Mac Studio M3 Ultra with 512 GB unified memory. Setup is llama.cpp b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the source rates are 80.93 t/s and 2778.95 t/s.

generation:

87.8 tokens/s

prompt processing (prefill):

2595.3 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 87.8 t/s generation and 2595.3 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999, running one GPU. The figures come from separate llama-bench phases: PP2048 at 2595.28 t/s and TG32 at 87.81 t/s, each the mean of five repetitions at context depth 0.

generation:

84.9 tokens/s

prompt processing (prefill):

2634.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 84.9 t/s generation and 2634.8 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with the UD-Q8_K_XL quant, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the article's llama-server, quality, VLM and long-context tests are separate.

generation:

88.1 tokens/s

prompt processing (prefill):

2554.2 tokens/s

quant:

UD-Q5_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 88.1 t/s generation and 2554.2 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with UD-Q5_K_M and F16 K and V cache, Metal backend, one GPU, -ngl 999, batch 2048, five repetitions. The figures are separate llama-bench phases: PP2048 at 2554.16 t/s and TG32 at 88.08 t/s, not a paired 2048-token prompt followed by 32 generated tokens.

generation:

62.4 tokens/s

prompt processing (prefill):

2649.3 tokens/s

quant:

BF16 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at 62.4 t/s generation and 2649.3 t/s prompt processing on an M3 Ultra Mac Studio with 512 GB of unified memory. Setup is llama.cpp build b8783 with BF16 weights, F16 K and V cache, Metal backend, and -ngl 999, using -p 2048 -n 32 -ub 2048 -fa 1. The figures are separate llama-bench phases: PP evaluates 2048 synthetic tokens and TG 32 sequential single tokens, with five repetitions and no speculative decoding.

generation:

14.9 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Llama 3.1 70B at 14.9 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX with 4-bit quantization, group_size 64, no speculative decoding or batching, at a 4k prompt (4138 actual tokens) with 128 output tokens. The figure is one observation, not a repeated-run mean, from a single timed generation.

generation:

14.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-72B-Instruct-4bit at 14.3 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX 0.31.1 with mlx-lm 0.31.2, 4-bit group_size=64 quantization, no speculative decoding or batching, at a 4132-token prompt with 128 output tokens. The figure is one observation, not a repeated-run mean; the source's more precise value is 14.331110981167 t/s, and the same sweep drops to 6.056028636 t/s at 131108 prompt tokens.

generation:

218.8 tokens/s

quant:

AXQ MXFP4 mixed (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Cyber-Tiel-Coder-35B-A3B at 218.8 t/s on an M3 Ultra with 512GB unified memory. Setup is the AX Engine with an AXQ MXFP4 trunk and protected MTP sidecar, one Metal GPU, expert streaming off, greedy decoding at MTP depth 3. The figure is the median decode rate over six samples on a 219-input/256-output python-lru workload, with a range of 217.75-220.11 t/s; completion including TTFT was 170.65 t/s.

generation:

6.8 tokens/s

quant:

Q8 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Llama 3.3 70B at 6.8 t/s on a Mac Studio M3 Ultra with 512GB unified memory. Setup is MLX with an 8-bit quantized mlx-community conversion, served over LAN by mlx_lm.server with temperature 0 and a 128-token output cap. The 6.8 t/s is the median of five measured requests at a nominal 32K context, with actual prompt median 32396 tokens; it is client-derived decode throughput, not native generation speed.

generation:

9.2 tokens/s

prompt processing (prefill):

295.9 tokens/s

quant:

BF16 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B at 9.2 t/s generation and 295.9 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with BF16 weights at 32768 input tokens and 128 output tokens, batch 1 on one device, with a fresh KV cache per call and no draft or KV-quant flags. The 32768-token row is a five-trial median, not pooled across lengths, with a peak allocator high-water of 71.714GB; a stuck indexer consumed one CPU core during the tests and its effect was not measured.