- reported speed:
- 18.3 tokens/s generation · 235.7 tokens/s prompt processing
- quant:
- oQ4e (MLX)
- kv:
- 8.0
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports slow performance with Qwen3.8-Flash-Next on an M2 Max 96GB using the oMLX engine. Benchmarks give 124.1 pp TPS and 20.4 tg TPS at pp1024/tg128, 170.7 pp TPS and 16.7 tg TPS at pp4096/tg128, 214.3 pp TPS and 18.5 tg TPS at pp8192/tg128, and 235.7 pp TPS and 18.3 tg TPS at pp16384/tg128.
Setup is oMLX, a custom MLX-based engine, with 32768 context and a turboquant_kv_bits 8.0 KV cache. The user also tried oQ4e-fp16-mtp, oQ3-fp16-mtp and oQ3-MTP variants.
The fp16 variants are even slower at roughly 150 pp TPS and 8 tg TPS. The user notes llama.cpp achieves 350-400 pp TPS. The model is MoE with 6B active parameters.
- reported speed:
- 43.0 tokens/s generation
- quant:
- Q4 (GGUF)
- kv:
- q8_0
- rating:
- 5/5
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
visioncoding
User reports a VLM is best overall for OCR and detail extraction, correctly reading mixed-script text (Chinese + Latin) and catching fine details other models missed.
Output is verbose at 1.4-2.2k tokens.
The user recommends it as the default for a coding-assistant MCP.
- reported speed:
- 8.6 tokens/s generation · 21.9 tokens/s prompt processing
- quant:
- IQ4_XS (gguf)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingvision
User reports Qwen3.8 27B IQ4_XS with vision projector on an M2 MacBook Pro 32GB, at 21.9 t/s prompt and 8.6 t/s generation.
Setup is llama.cpp built from source, with context set to 128K due to memory constraints.
User notes the model is slow on Mac but that thinking quality is good, and mentions previous experience with Qwen3.5 35B-A3B.
- reported speed:
- 22.4 tokens/s generation · 360.2 tokens/s prompt processing
- quant:
- Q6_K_XL (gguf)
- kv:
- Q8
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports llama-bench results and a serving setup, with about 16 t/s on WebUI with a specific prompt.
- reported speed:
- 25.8 tokens/s generation · 250.0 tokens/s prompt processing
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a custom llama.cpp fork on an M2 Ultra with a repacked model at 141 GiB, smaller than Q4 GGUF, peaking at 42 t/s generation.
Setup uses an SSD KV cache and dynamic lanes, 8 lanes.
Prompt processing runs about 250 t/s at 8k-32k context.
- reported speed:
- 22.0 tokens/s generation · 109.0 tokens/s prompt processing
- quant:
- Q8_0 (gguf)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagentic
User benchmarks Qwen3.8-27B on an Apple M2 Max 96GB, with MTPLX reaching 20-22 t/s decode and 109 t/s prefill.
Setup is 8-bit quants in GGUF and MLX with an unquantised KV cache at 262k context, across llama.cpp baseline, llama.cpp+MTP, MTPLX, oMLX, mlx-dspark (DFlash2/DSpark) and vllm-mlx, on an agentic coding benchmark with 4 phases.
MTPLX xhigh is worth the extra time, while oMLX xhigh is not. vllm-mlx leaks chain-of-thought into the output, and mlx-dspark DFlash2 xhigh thought 226K tokens and delivered nothing.
- reported speed:
- 28.0 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports decode speeds at various context depths: 28 t/s at the start, 23.5 t/s at 45k, and 18 t/s at 192k.
The run was maintained with 8k token output.
Prefill performance is mentioned but no numbers are given.
- reported speed:
- 5.5 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports 31-35 t/s on an M5 MacBook Pro with a custom Swift/Metal engine.
Setup is an OpenAI-compatible server with streaming and tool-call support.
- quant:
- Q4_K_M (gguf)
vision
User reports a vision benchmark champion model at 70 s/img on an Apple M2 Max 96GB with llama.cpp b9690.
The model gives the best quality and stability with thinking disabled, and completed 90/90 successful runs.
- reported speed:
- 8.0 tokens/s generation
- quant:
- F16 (gguf)
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagentic
User benchmarks Qwen 3.6 27B F16 on an M2 Max 96GB with llama.cpp and MTP speculative decoding, reporting 8-18 tok/s depending on the task.
Without MTP the same setup reaches 6.6 tok/s. A Q8 quant was also tested but performed worse.
Context up to 150k+ tokens is usable. The user notes chat template fixes were critical, and the setup was used for agentic coding to create a Pacman game.
- reported speed:
- 28.0 tokens/s generation
- quant:
- Q5_K_M (gguf)
- kv:
- Q4
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingagentic
User reports MTP speculative decoding gives a 2.5x speedup on an M2 Max 96GB.
Setup is Q5_K_M quant with a q4_0 KV cache.
The user also provides hardware recommendations for various Apple Silicon and NVIDIA GPUs.