M2 Max 96GB reports

generation:

20.1 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B 4-bit on an M2 Max 96GB, comparing mlx-serve with MTP speculative decoding against LM Studio. With the MTP head correctly loaded, mlx-serve reaches 20.1 t/s decode on long prompts and 13.0 t/s on short, versus 11.2 and 8.0 t/s without speculation. Setup is mlx-serve 26.9.5 with MLX 0.32.2 on macOS 26.6.2, using the mlx-community Qwen3.8-27B-MTP-4bit sidecar. Prefill measured 49/145 tok/s with MTP loaded and 47/112 tok/s without. LM Studio on the same weights reached 11.8 t/s decode short and 11.6 t/s long, with 59/101 tok/s prefill. The user reports that mlx-serve silently falls back to Prompt Lookup Decoding when the MTP sidecar tensors lack the required mtp. prefix, and that MoE MTP heads fail to load entirely, so Qwen3.6-35B-A3B got no speedup. Numbers are client-side medians of 5 runs with unique prefixes to avoid prefix caching.

generation:

18.3 tokens/s

prompt processing (prefill):

235.7 tokens/s

quant:

oQ4e (MLX)

kv:

8.0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports slow performance with Qwen3.8-Flash-Next on an M2 Max 96GB using the oMLX engine. Benchmarks give 124.1 pp TPS and 20.4 tg TPS at pp1024/tg128, 170.7 pp TPS and 16.7 tg TPS at pp4096/tg128, 214.3 pp TPS and 18.5 tg TPS at pp8192/tg128, and 235.7 pp TPS and 18.3 tg TPS at pp16384/tg128. Setup is oMLX, a custom MLX-based engine, with 32768 context and a turboquant_kv_bits 8.0 KV cache. The user also tried oQ4e-fp16-mtp, oQ3-fp16-mtp and oQ3-MTP variants. The fp16 variants are even slower at roughly 150 pp TPS and 8 tg TPS. The user notes llama.cpp achieves 350-400 pp TPS. The model is MoE with 6B active parameters.

generation:

43.0 tokens/s

quant:

Q4 (GGUF)

kv:

q8_0

rating:

5/5

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a VLM is best overall for OCR and detail extraction, correctly reading mixed-script text (Chinese + Latin) and catching fine details other models missed. Output is verbose at 1.4-2.2k tokens. The user recommends it as the default for a coding-assistant MCP.

generation:

20-22 tokens/s

prompt processing (prefill):

109.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on an Apple M2 Max 96GB, with MTPLX reaching 20-22 t/s decode and 109 t/s prefill. Setup is 8-bit quants in GGUF and MLX with an unquantised KV cache at 262k context, across llama.cpp baseline, llama.cpp+MTP, MTPLX, oMLX, mlx-dspark (DFlash2/DSpark) and vllm-mlx, on an agentic coding benchmark with 4 phases. MTPLX xhigh is worth the extra time, while oMLX xhigh is not. vllm-mlx leaks chain-of-thought into the output, and mlx-dspark DFlash2 xhigh thought 226K tokens and delivered nothing.

quant:

Q4_K_M (GGUF)

User reports a vision benchmark champion model at 70 s/img on an Apple M2 Max 96GB with llama.cpp b9690. The model gives the best quality and stability with thinking disabled, and completed 90/90 successful runs.

generation:

8.0 tokens/s

quant:

F16 (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen 3.6 27B F16 on an M2 Max 96GB with llama.cpp and MTP speculative decoding, reporting 8-18 tok/s depending on the task. Without MTP the same setup reaches 6.6 tok/s. A Q8 quant was also tested but performed worse. Context up to 150k+ tokens is usable. The user notes chat template fixes were critical, and the setup was used for agentic coding to create a Pacman game.

generation:

28.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MTP speculative decoding gives a 2.5x speedup on an M2 Max 96GB. Setup is Q5_K_M quant with a q4_0 KV cache. The user also provides hardware recommendations for various Apple Silicon and NVIDIA GPUs.