M2 Max 96GB

reported speed:

18.3 tokens/s generation · 235.7 tokens/s prompt processing

quant:

oQ4e (MLX)

kv:

8.0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports slow performance with Qwen3.8-Flash-Next on an M2 Max 96GB using the oMLX engine. Benchmarks give 124.1 pp TPS and 20.4 tg TPS at pp1024/tg128, 170.7 pp TPS and 16.7 tg TPS at pp4096/tg128, 214.3 pp TPS and 18.5 tg TPS at pp8192/tg128, and 235.7 pp TPS and 18.3 tg TPS at pp16384/tg128. Setup is oMLX, a custom MLX-based engine, with 32768 context and a turboquant_kv_bits 8.0 KV cache. The user also tried oQ4e-fp16-mtp, oQ3-fp16-mtp and oQ3-MTP variants. The fp16 variants are even slower at roughly 150 pp TPS and 8 tg TPS. The user notes llama.cpp achieves 350-400 pp TPS. The model is MoE with 6B active parameters.

reported speed:

43.0 tokens/s generation

quant:

Q4 (GGUF)

kv:

q8_0

rating:

5/5

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a VLM is best overall for OCR and detail extraction, correctly reading mixed-script text (Chinese + Latin) and catching fine details other models missed. Output is verbose at 1.4-2.2k tokens. The user recommends it as the default for a coding-assistant MCP.

reported speed:

22.0 tokens/s generation · 109.0 tokens/s prompt processing

quant:

Q8_0 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on an Apple M2 Max 96GB, with MTPLX reaching 20-22 t/s decode and 109 t/s prefill. Setup is 8-bit quants in GGUF and MLX with an unquantised KV cache at 262k context, across llama.cpp baseline, llama.cpp+MTP, MTPLX, oMLX, mlx-dspark (DFlash2/DSpark) and vllm-mlx, on an agentic coding benchmark with 4 phases. MTPLX xhigh is worth the extra time, while oMLX xhigh is not. vllm-mlx leaks chain-of-thought into the output, and mlx-dspark DFlash2 xhigh thought 226K tokens and delivered nothing.

quant:

Q4_K_M (gguf)

User reports a vision benchmark champion model at 70 s/img on an Apple M2 Max 96GB with llama.cpp b9690. The model gives the best quality and stability with thinking disabled, and completed 90/90 successful runs.

reported speed:

8.0 tokens/s generation

quant:

F16 (gguf)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks Qwen 3.6 27B F16 on an M2 Max 96GB with llama.cpp and MTP speculative decoding, reporting 8-18 tok/s depending on the task. Without MTP the same setup reaches 6.6 tok/s. A Q8 quant was also tested but performed worse. Context up to 150k+ tokens is usable. The user notes chat template fixes were critical, and the setup was used for agentic coding to create a Pacman game.

reported speed:

28.0 tokens/s generation

quant:

Q5_K_M (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports MTP speculative decoding gives a 2.5x speedup on an M2 Max 96GB. Setup is Q5_K_M quant with a q4_0 KV cache. The user also provides hardware recommendations for various Apple Silicon and NVIDIA GPUs.