Mac M3 LLM performance

reported speed:

7.4 tokens/s generation · 121.3 tokens/s prompt processing

quant:

8bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 27B at 7.4 t/s generation and 121.3 t/s prompt processing on an M3 Max 96GB. Setup is oMLX with the MLX 8-bit model at pp1024/tg128, using 28.34 GB peak memory. A pp4096/tg128 run reached 8.8 t/s generation and 133.8 t/s prompt processing. Continuous batching at 4x reached 19.9 t/s aggregate generation. User is new to LLMs and asks whether the slow speed is due to the dense model or a setup problem.

reported speed:

19.7 tokens/s generation · 439.4 tokens/s prompt processing

quant:

oQ4e

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash on an M3 Ultra 512GB with oQ4e and Engram in RAM, reaching 439.4 t/s prefill and 19.7 t/s generation at 64K context with MTP off, and 435.6 t/s prefill and 39.7 t/s generation with MTP on. Setup is oMLX 0.7.0.dev2 with Python code prompts, temperature 1.0, 128 generated tokens, and no prefix cache. One measured run per configuration after warm-up. At 4K context, prefill was 458.0 t/s with MTP off and 452.2 t/s with MTP on, while generation was 20.2 t/s and 32.1 t/s respectively. At 16K, prefill was 459.1 t/s and 454.8 t/s, with generation at 20.0 t/s and 34.7 t/s. At 32K, prefill was 452.2 t/s and 447.7 t/s, with generation at 19.8 t/s and 31.5 t/s. The user also notes experimental MoE Expert SSD Offload support for DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE, and an M5 prefill speedup from 615.2 to 826.7 t/s at 32K on an M5 Max with Qwen3.8-27B using INT8-activation kernels.

reported speed:

17.3 tokens/s generation

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a bug in oMLX 0.6.4 distributed clustering where the coordinator fails to release RAM and GPU after a crash. The model is mlx-community/MiniMax-M3-4bit at 236 GB, run across rank 0 on a Mac Studio M3 Ultra 256GB and rank 1 on a Mac Studio M2 Ultra 192GB. The first completion produced 17 tokens from a 7,693-token prompt at 17.3 tok/s. The crash was triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching. After the crash, roughly 116 GB of wired memory has no owning process and the GPU stays pinned at 100%, requiring a reboot. The M2 Ultra worker released memory cleanly. The user also reports that MiniMax-M3 crashes the cluster after the first prompt.

reported speed:

38.0 tokens/s generation

quant:

T5 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a custom oMLX fork with a Metal kernel for ternary experts running at 30.7 t/s on a 65,536-token prompt plus 256 output tokens, with prefill at 210-380 tok/s. Setup is a ternary routed expert gate/up (Bonsai T5 packing, ~1.875 bpw), Q3 expert down projections, and a 53 GB n-gram table left on SSD as Q8 and mmapped per token, all low-bit tensors fitted with Unsloth's imatrix. Stock oMLX/mlx-lm will not load it. 64K context is confirmed; 96K trips the prefill guard. Physical peak is 42.3 GiB at 64K and ~41.5 GiB at 8K, with ~35.6 GiB resident and ~2 GiB swapped once at load. The oMLX memory guard is on the 'safe' profile with a 48 GB limit, one model and one request at a time. Against Unsloth UD-Q4_K_XL, which does not fit in 48 GB, the user reports KLD vs Q8_0 of 0.49 vs 0.036, MMLU 83.0% vs 89.7%, GSM8K 90.0% vs 92.0%, and HumanEval 92.7% vs 95.7%. The model sometimes ignores 'answer with just the letter' in Chinese.

reported speed:

37.4 tokens/s generation · 550.0 tokens/s prompt processing

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports GLM-5.3-Flash on an M3 Ultra at 60 t/s for SQL and 38 t/s average with an agent harness. Setup uses a dflash drafter for speculative decoding, with prefill at 550 t/s at 62k context and generation at 37.4 t/s at 300k context. The user states accuracy is preserved.

reported speed:

12.7 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 11-13 t/s on an M3 Max 96GB with SSD streaming and iogpu.wired_limit_mb=86016. Setup is antirez's ds4 engine with GGUF. TTFT is 3-5s after warmup. Prefill of 36k tokens takes about 2.5 minutes.

reported speed:

20.0 tokens/s generation · 533.0 tokens/s prompt processing

quant:

8bit (affine)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports an optimized DeepSeek V4 Flash 8-bit affine MLX model on oMLX, with prefill improving from ~300-321 to ~533 tok/s and decode from ~7.31 to ~20-22 tok/s. Real runs at 79K-119K context show 19.2-20.7 tok/s. User asks for community review on accuracy and next optimization directions.

reported speed:

8.0 tokens/s generation

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 8 t/s on a Mac Studio M3 Ultra 512GB with DeepSeek V4 Flash GGUF Q4_K_XL. The user also tried Q8. The user expected better performance.

reported speed:

43.0 tokens/s generation

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 34 t/s at the start of a run, rising to 43 t/s by the end. The cached tokens were the default chat prompt and the query was 13k.

reported speed:

475.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4 Flash serving on an M3 Ultra 512GB with the ds4 engine, with cold prefill improved from 392 to 475 t/s at 64k context through kernel optimizations. Cache prewarming with max_tokens:0 yields roughly 10x speedup for chat turns, from 6-20s down to 1.6s.

reported speed:

17.7 tokens/s generation

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a custom C + Metal runtime with speculative decoding at 18 t/s end-to-end on coding tasks and 17.7 t/s for an LRUCache implementation, on 36 GB unified memory. Q4 weights are about 15 GB, mmap'd. Prose runs at roughly 10-11 t/s. TTFT is about 1.4s for a short prompt and about 2.7s for a 128-token prompt. The user compares the custom runtime against llama.cpp and reports it faster.

quant:

UD-IQ4_XS

User compares Mac Studio configs (M5 Ultra 96GB, M5 Max 128GB, M5 Max 64GB) for local LLM use. They discuss Qwen3.8-27B at Q8 and Qwen3.8-Flash-Next (6B active) at various quants, including Flash-Next on 96GB Ultra with UD-IQ4_XS (93.7GB, 91.1% retention) and on 128GB Max with UD-Q4_K_XL (111GB, 93.5% retention), with the N-gram layer offloaded to SSD. User asks for advice on bandwidth versus quant tier, and whether to just get the 64GB box. No actual benchmark numbers are reported.

quant:

4bit

User runs Qwen3.6-35B-A3B-4bit on an M3 Max 128GB for production sub-agent delegations. User also mentions GLM-5.1 for orchestration. User is considering building a 5090 rig.

reported speed:

5.5 tokens/s generation · 160.0 tokens/s prompt processing

quant:

Q8 (mlx)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 160 t/s prefill and 5-6 t/s generation on an M5 Max 128GB with Qwen 3.6 27B Q8 MLX at 290k context. GPU utilization sits at only 36-50%. User expected 8-14 t/s generation and asks how other setups compare.