Mac M5 LLM performance

reported speed:

133.6 tokens/s generation

quant:

MLX-4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Nex-N2.5-mini at 133.6 t/s on an Apple M5 Max. Setup is MLX with the MLX-4bit quant, using temperature 0.7, top_p 0.95, top_k 40, and reasoning_effort high. User calls the generation speed and prompt processing really good, the memory footprint okay, and the quality good across the board, and plans to use it for agents and possibly coding.

reported speed:

24.0 tokens/s generation · 333.0 tokens/s prompt processing

quant:

IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks GLM-5.3 321B across a MacBook M5 Max 128GB and an AMD Strix Halo 128GB connected over TB4. The default tensor split was slower than the Mac alone. A Mac-heavy split improved prompt processing from 188 to 333 t/s and generation from 17 to 24 t/s. User also reports 12-15 t/s for model sizes of 148-200 GB and plans to add two Asus GX10 next week.

reported speed:

9.0 tokens/s generation

quant:

Q3_xxs

kv:

8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User asks whether about 9 t/s is normal for their setup, and is new to local LLMs. User also asks for model recommendations for PDF summaries on 16 GB of RAM.

reported speed:

55.0 tokens/s generation

quant:

oQ4e

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User asks about a Mac Studio M5 Max 128GB for local Qwen3.8-flash-next. The user mentions the oQ4e quant at ~55 tok/s and also the oQ5e quant. The user is debating whether to buy now or wait for the M7 Ultra, for private documents, notes, coding and general assistant work.

reported speed:

40.0 tokens/s generation

quant:

mixed-4-8bit

kv:

8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports about 40 t/s on prose and 75 t/s on coding at roughly 760k context. Setup uses 8-bit dense layers and 4-bit expert layers, with peak memory around 117 GB and a requirement of iogpu.wired_limit_mb=120000. Model weights are at ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit.

reported speed:

17.2 tokens/s generation · 706.0 tokens/s prompt processing

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks a 128 GB M5 Max MacBook Pro across multiple models, selecting Qwen 3.6 27B MLX Q8 as the primary one. Other models tested include Gemma 4, MiniMax M2.7 and DeepSeek V4 Flash. The user is enthusiastic about local AI.

reported speed:

79.4 tokens/s generation · 2153.0 tokens/s prompt processing

quant:

Q8 XL

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks several models on an M5 Max 128GB and selects Qwen3.6 35B A3B as the primary model for its MoE detail. Other models tested include Gemma 4, Qwen 3.6 27B, MiniMax M2.7, Mistral Medium 3.5, Step 3.7 Flash and DeepSeek V4 Flash. Results show strong performance for MoE models.

quant:

IQ2_XXS/Q2_K (GGUF)

User compares DeepSeek-V4-Flash on an M5 Max MacBook with 128 GB against 2x DGX Spark. Terminal-Bench 2.1 scores 54% (47/87) on the Mac and 52% (45/86) on the Spark pair. The Mac runs a heavily quantized GGUF at IQ2_XXS/Q2_K, about 2.45 bits per weight. The Spark pair runs the native FP8/FP4 checkpoint. The Mac wins 11 tasks outright and the Spark pair 9, with no significant difference in task-solving. The Spark pair offers higher speed at 54-58 tok/s single-stream and 253 tok/s aggregate at concurrency 8, plus a larger 262K server context window.

reported speed:

30.0 tokens/s generation

quant:

Q2-Q4 mixed imatrix (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports about 30 t/s on an M5 Max with a quantized DeepSeek V4 Flash running on the DS4 engine. The user states this is double the speed of llama.cpp. The user asks for feedback from CUDA and ROCm users.

reported speed:

1.0 tokens/s generation · 50.0 tokens/s prompt processing

quant:

4bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at about 50 t/s prefill and about 1 t/s decode on an M5 Air with 32 GB. Setup uses the streamed experts trick with a model of roughly 300B total parameters.

reported speed:

44.0 tokens/s generation

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares DeepSeek V4 Flash 9B distill against its base model Qwen3.5 9B on a MacBook Pro M5 Pro with 48 GB RAM. Both models gave the same answers on 6/8 tasks, but the distill used fewer tokens (5480 vs 8975). Throughput was near identical (44 vs 41 tok/s). The distill was slower on open-ended tasks (1467 tokens vs 867 for a buffer overflow explanation). The base model failed arithmetic due to a token cap.

reported speed:

15.0 tokens/s generation

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 10-17 t/s in SSD streaming mode. Setup keeps routed experts partly in RAM cache and pulls them from the GGUF on cache misses. Experts and output head are Q8_0, while router, embeddings and V4 auxiliary blocks are FP16.

reported speed:

31.8 tokens/s generation

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on an M5 Max 128GB across four engines. Setup is llama.cpp with native MTP at 31.8 t/s sustained and 22.7 t/s at 56k context, TTFT 125ms, prefix reuse 0.80, RAM 36.8GB; oMLX 0.6 at 29.8 t/s and 24.6 t/s at depth, TTFT 1968ms, prefix reuse 0.82, ~17GB; mlx-vlm with MTP at 43.3 t/s and 28.9 t/s at depth, TTFT 9982ms, no prefix reuse, 15.6GB; and MTPLX 2.6 at 27.1 t/s and 23.1 t/s at depth, TTFT 13400ms, no prefix reuse. The fastest engine, mlx-vlm, is unusable for agents due to 10s TTFT and no prefix reuse. Without --spec-type draft-mtp, llama.cpp drops the MTP head; a draft cap of 4 gives 31.8 t/s versus 21.9 t/s with no speculation, cap 2 from Qwen3.6 presets gives only 22.2 t/s, and cap 6 loses gains. At 262k context with q8_0 KV cache llama.cpp gives 29.6 t/s and 18.5 t/s at depth; at 131k with f16 it gives 31.8 t/s and 22.7 t/s at depth. The model does 5 searches then calls more tools instead of writing the brief and dies at max_turns. reasoning_effort 'none' throws an exception on 3.8, so enable_thinking: false is needed; oMLX does not activate MTP on 3.8; the model field of the response should be checked rather than /v1/models; and the M5 Max throttles to 50% after 80-110s. No generational verdict versus 3.6 is given.

reported speed:

10.5 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User asks which model to use for agentic coding. User reports DeepSeek V4 Flash at 10-11 t/s with 1M context, and Qwen3.8 27B at 15-16 t/s with FP8 quants.

reported speed:

2.0 tokens/s generation

quant:

mixed 4/8-bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 2.04 t/s with an 8 GB expert cache and 1.23 t/s with a 32 GB expert cache, streaming from SSD. Speculative decoding and prefetch both decreased throughput. Codebook 2-bit quantization failed the quality gate.

reported speed:

11.5 tokens/s generation · 333.0 tokens/s prompt processing

quant:

UD-Q2_K_XL (gguf)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 79 GB in 2-bit running at 350K context for 3.5 hours on a 128 GB M5 Max, over 100 turns. Prefill runs 1,561 t/s at 5.6K context tapering to 318 t/s at 111K, with normal turns at 77-854 t/s from prefix reuse. Decode is about 30-35 t/s at small context down to 11.5 t/s at 169K, with a dip to 7.7 t/s at around 140K due to Low Power Mode. Decode figures are effective throughput with speculative decoding. The user calls the experience strong for the first ~100K context, then reports role confusion on long-tail tasks, suspecting the 2-bit quant or preview-model long-context quality.

reported speed:

36.6 tokens/s generation

quant:

oQ4e-mtp

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8-27B MTP quants on an Apple M5 Max. The oQ4e-mtp quant reaches 36.6 t/s with a quality score of 86.8. User calls oQ4e-mtp the best balance of speed and quality and recommends it as the sweet spot.

reported speed:

19.0 tokens/s generation · 325.0 tokens/s prompt processing

quant:

Q8_0 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares MLX and GGUF for Qwen3.8 27B on an M5 Pro, reporting generation at about 19 t/s. Setup is llama.cpp with Metal, which the user says now matches MLX prefill at roughly 300-350 t/s. The user questions whether MLX is still needed and mentions MTP and oMLX.

reported speed:

33.9 tokens/s generation

quant:

Q4_K_S (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks prompt processing at 65,536 context, with throughput falling from 62.80 t/s at 128 context to 33.92 t/s at 65,536. Memory peaked at ~120+ GB, and the system stayed sluggish but usable.

reported speed:

32.4 tokens/s generation

quant:

UD128 (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Tencent HY3 295B-A21B MoE at 32.4 t/s generation at empty context and 16.3 t/s at 16K context on Metal. Setup is llama.cpp with the UD128 quant (107 GB), a q8_0 KV cache, and 24K context. With MTP n=2, generation peaks at 38 t/s. Prefill runs 528 t/s at pp512 empty and 124 t/s at pp512 at 16K. User compares the result favorably to DeepSeek V4 Flash.