M3 Ultra 256GB

generation:

4.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Spark Open 816B-A42B decoding at 4 t/s on an M3 Ultra 256GB. Setup is ds4; no quant, context length or KV cache is stated. The user calls this the first successful run that reached decode and hopes for a 4-5x improvement with larger prefill chunks.

generation:

77.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 77.8 t/s generation. Source evidence (verbatim): Source: https://terminalbytes.com/ollama-vs-llama-cpp-vs-mlx-mac-2026/ # Ollama vs llama.cpp vs MLX on a Mac Studio, measured Hardware: Mac Studio with an M3 Ultra and 256GB unified memory. The author ran Qwen3.5 9B through every runtime with the same 34-token prompt, thinking off, one request at a time, two runs each, no speculative decoding. Results, generation speed: llama.cpp llama-bench build 10809 with unsloth Qwen3.5-9B Q4_K_M GGUF, tg128 77.8 tok/s; Ollama 0.32.13 GGUF qwen3.5:9b Q4_K_M 77.7 tok/s; Ollama 0.32.13 MLX NVFP4 90.5 tok/s; rapid-mlx 0.14.1 Qwen3.5-9B 4-bit 105.9 tok/s; mlx-lm 0.31.3 Qwen3.5-9B 4-bit 107.5 tok/s. The author wrote down the numbers from his own runs.

generation:

51-59 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next decoding at a median of 51-59 tokens/s (p90 ~65) on two M3 Ultra 256GB Mac Studios. Setup is MTPLX with a 4-bit mixture-of-experts model (107 GB on disk) and multi-token prediction, carrying 116-142k-token prompts with a 97-99% prompt-cache hit rate. The user also compares DeepSeek V4.1 Flash against MiMo V2.6 Flash head to head (8 agents each, 3 hours): 11 vs 7 PRs merged and 1 vs 4 stalls. DeepSeek costs about 1.6x per output token but less per merged PR, so every builder now runs DeepSeek.

generation:

77.8 tokens/s

prompt processing (prefill):

1010.3 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 77.8 t/s generation and 1010.3 t/s prompt processing on an M3 Ultra 256GB. Setup is llama.cpp with Q4_K_M, run through native llama-bench with -p 512 -n 128 -r 3. The 77.83 ± 0.11 t/s and 1010.32 ± 1.28 tokens/s are native benchmark rates for 512/128 test tokens, not configured maximum context; KV precision, flash attention and GPU-offload count are unreported.

generation:

73.1 tokens/s

quant:

oQ5 affine mixed precision, group 64 (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ling-3.0 Flash at 73.1 t/s on an M3 Ultra 256GB. Setup is oMLX with oQ5 affine mixed precision, group 64, on a 124B total / 5.1B active model. The figure is the median of three greedy 256-token runs (73.23, 73.10, 73.09) after a 256-token warm-up, with a 39-token prompt; peak memory is reported at 87.44 GB.

generation:

48.8 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash Abliterated at 48.8 tok/s decode on a Mac Studio M3 Ultra 256 GB. Setup is mlx-serve 26.10.1 with MLX 4-bit weights and 131072 context, MTP on with draft rerank, resident size about 172–181 GB. Real sessions at 50k–85k tokens decode about 45–63 tok/s with prefill 150–300 tok/s on cache hits. A short probe at 8192 context measured 49.9 tok/s. TensorFold 1.0.5 on the same Mac left the MTP head on disk and one short reply measured 15.7 tok/s.

generation:

14.0 tokens/s

prompt processing (prefill):

93.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 14.0 tok/s generation and 93.1 tok/s prompt processing on a Mac Studio M3 Ultra with 256GB unified memory. Setup is Ollama with Q4_K_M (17GB) and five timed runs averaged; the predecessor qwen3.6:27b reached 28.6 tok/s on the same machine, and the 1-bit UD-IQ1_M quant (6.7GB) ran at 27.2 tok/s generation and 309 tok/s prompt processing in llama.cpp. User notes the new hybrid attention architecture is slower per token but uses roughly a third the tokens per answer, so wall-clock per answer is close to a tie; community figures cited for Strix Halo (20.4 tok/s), dual RTX 3090 (~60 tok/s) and RTX 5090 (75-140 tok/s) are not the author's own measurements.

generation:

43.4 tokens/s

quant:

4bit (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 43.38 t/s generation on a Mac Studio M3 Ultra 256GB, the median of three requests with an 8,156-token prompt. Setup is Rapid-MLX 0.13.4 with the rapid-mlx MLX 4-bit MTP checkpoint, one request, with the engine's adaptive MTP on by default for this checkpoint. The source's 330.8 t/s prefill is the prompt divided by time to first token, so no prompt speed is stored. The figures come from the Rapid-MLX benchmark docs, not an independent rerun.

generation:

107.5 tokens/s

quant:

MLX 4-bit affine (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 107.5 t/s generation on an Apple Mac Studio with M3 Ultra and 256 GB unified memory. Setup is mlx-lm 0.31.3 with the mlx-community 4-bit MLX quant, thinking off and no speculative decoding; peak memory was 5.24 GB. The figure comes from a 34-token request that generated 1,156 tokens for a 900-word essay; the author ran two passes and reports variation of about 1 t/s or less. Configured context length is not stated for this run.

generation:

25.7 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 25.67 t/s on an M3 Ultra 256GB. Setup is Rapid-MLX 0.13.2 with mixed 4-bit affine MLX quantization and a 2048-token prefill chunk, at roughly 102.8-103.8GB active unified allocation. Decode falls to 24.27 / 23.40 / 21.72 t/s at the 2048 / 8192 / 32768 prompt targets, with median TTFT of 2.262 / 9.236 / 44.659 seconds. Each row is the median of three batch-size-one runs with cold prefix cache and 256 requested decode tokens.

generation:

84.1 tokens/s

prompt processing (prefill):

285-325 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B at 84.1 and 72.1 tok/s on production-shaped prompts and 87.0 tok/s at 12K context on a Mac Studio M3 Ultra 256GB. Setup is mlx-serve 26.9.6-dev with a 4-bit MTP pack and native MTP; prefill measured at roughly 285-325 tok/s, and concurrency gave no gain (1.01x at N=4). The same model class under Inco Splash reached 85.3-85.7 tok/s on short prompts and 65.7 tok/s at 32K, scaling 1.18x at N=2 and 1.27-1.29x at N=4 before going flat, with the engine reporting a maximum batch width of 4. The user notes the two engines' single-stream figures came from different prompts, so the 27B single-stream comparison is left undecided. Also measured: Qwen3.6-35B-A3B with mlx-dspark at accept length 4.5998 and 2.33x, versus MTPLX at accept 2.2294 and about 1.47x; Qwen3.8-Flash-Next with mlx-serve 26.9.6 and --mtp at 76.8 tok/s single stream (+32% over the same build without the flag) and 133.5 tok/s aggregate at N=8.

generation:

25.0 tokens/s

quant:

4bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Minimax-m2.1-4bit at 25 t/s generation on an M3 Ultra 256GB with 80-core GPU, compared against llama.cpp with flash attention at 32 t/s. Setup is MLX with 4bit weights at 30K context; the same model under llama.cpp with flash attention reached 32 t/s generation and 78s prompt processing. At 146K context MLX fell to 5.95 t/s generation versus 12.12 t/s for llama.cpp, and the user says MLX runs about 50% slower on long-context generation, impacting agentic coding.

generation:

22.0 tokens/s

prompt processing (prefill):

75.0 tokens/s

quant:

Q4Q8 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at ~22 tok/s decode and ~75 tok/s prefill on a Mac Studio M3 Ultra with 256GB unified memory. Setup is vMLX with a mixed-precision MLX affine quantization (Q4 routed experts, Q8 attention and shared expert, fp16 norms) at group size 32, 173 GB on disk, 1,048,576 token context. vMLX disables continuous batching for this model, so all requests go through SimpleEngine; the bundle requires a runtime patch to vMLX's jang_tools loader to avoid 8-bit expert wrapping and BOS-token loops.

generation:

17.3 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a bug in oMLX 0.6.4 distributed clustering where the coordinator fails to release RAM and GPU after a crash. The model is mlx-community/MiniMax-M3-4bit at 236 GB, run across rank 0 on a Mac Studio M3 Ultra 256GB and rank 1 on a Mac Studio M2 Ultra 192GB. The first completion produced 17 tokens from a 7,693-token prompt at 17.3 tok/s. The crash was triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching. After the crash, roughly 116 GB of wired memory has no owning process and the GPU stays pinned at 100%, requiring a reboot. The M2 Ultra worker released memory cleanly. The user also reports that MiniMax-M3 crashes the cluster after the first prompt.

generation:

37.4 tokens/s

prompt processing (prefill):

550.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash Q4 at 37.4 t/s end to end at 300k context on an M3 Ultra, against 21.6 t/s for stock ds4. Setup is a ds4 branch tuned for the M3 Ultra, decoding serially in a single stream without the dflash drafter, with prefill at 550 t/s at 62k. Serial decode rises from 29 to 40 t/s at short context and from 24 to 38 at 62k. In a claude code harness at ~200k depth it averages over 38 t/s output and reaches 60 t/s writing SQL, where the optional dflash drafter adds +20 to +50 percent.