Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 27B 4-bit on an M2 Max 96GB, comparing mlx-serve with MTP speculative decoding against LM Studio. With the MTP head correctly loaded, mlx-serve reaches 20.1 t/s decode on long prompts and 13.0 t/s on short, versus 11.2 and 8.0 t/s without speculation.
Setup is mlx-serve 26.9.5 with MLX 0.32.2 on macOS 26.6.2, using the mlx-community Qwen3.8-27B-MTP-4bit sidecar. Prefill measured 49/145 tok/s with MTP loaded and 47/112 tok/s without. LM Studio on the same weights reached 11.8 t/s decode short and 11.6 t/s long, with 59/101 tok/s prefill.
The user reports that mlx-serve silently falls back to Prompt Lookup Decoding when the MTP sidecar tensors lack the required mtp. prefix, and that MoE MTP heads fail to load entirely, so Qwen3.6-35B-A3B got no speedup. Numbers are client-side medians of 5 runs with unique prefixes to avoid prefix caching.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 4B at 70.4 t/s on an M2 Max using native GGUF support in transformers.
Setup is the Q4_K_M GGUF from unsloth, reusing ggml kernels on Apple Silicon. llama.cpp reached 71.8 t/s on the same model.
Two other models were also tested: Qwen3.8 27B UD-Q4_K_M at 15.9 t/s (vs 13.4 t/s with llama.cpp) and Qwen3.5 35B-A3B UD-IQ4_XS at 60.2 t/s (vs 61.3 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingvisionlong-context
User reports Bonsai 2 27B at 7.6 t/s generation and 42.0 t/s prompt processing on an 8GB M2 Mac mini, with all layers offloaded to the GPU and zero swap.
Setup is PrismML's llama.cpp fork (prism-b10683-d8f26ee) with a ternary 5.95GB GGUF, flash attention on, -ub 256, q4_0 KV cache, -np 1, and iogpu.wired_limit_mb=6656. The default -ub 512 and f16 KV both fail to allocate; -ub 128 with f16 KV also runs at identical speed. Setting the wired limit to 7168 kernel-panicked the machine.
In real use with llama-server at 8k context, a 1,896-token prompt took 47s to first token (pp ~40 t/s) and generated at 7.0 t/s; the same prompt via prefix cache took 0.8s. 16k context works but 32k OOMs, and PQ2_0 (7.21GB) does not fit. The user notes greedy decoding falls into repetition loops and recommends temp 1.0 / top-p 0.95 / top-k 20.
User benchmarks quantization accuracy damage on Qwen2.5-1.5B and 0.5B across the Q8 to Q2 ladder, running 24,600 generations on an 8GB M2 Air with llama.cpp and MLX.
On the 1.5B model, JSON schema conformance stays at 100% at every rung including Q2, GSM8K arithmetic falls from 56.5% to 19.0% with the first significant drop at Q3, and TriviaQA recall falls from 19% to 9%, breaking only at Q2.
Q4 significantly beat Q8 on the 1.5B model in both runtimes (p=0.044 llama.cpp, p=0.015 MLX), while on the 0.5B model MLX 4-bit collapses from 45% to 27.5%. Q2_K is strictly dominated on 0.5B: Q3 gives 40.5% at 122 tok/s versus Q2 at 13.5% and 119 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.6 35B A3B at 15-20 t/s on an M2 Max 32GB, with prompt processing taking 20 minutes for 40k tokens when the model fits in RAM.
Setup is Q4_XS GGUF at 60,000 context with a Q8 KV cache.
With SSD streaming the model runs at 4-10 t/s and prompt processing takes 50-60 minutes for 40-60k tokens. User asks whether to upgrade to an M1 Max 64GB, M3 Max 128GB, or a 24GB GPU build with 128GB RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4.1 Flash at 1.8-2.2 t/s on an M2 Max 64GB.
Setup is the Whallm app with MLX, loading parts of the model from SSD, around 33 GiB peak MLX memory, with 1K-16K input tokens.
The app also adds built-in throughput benchmarks and an OpenAI-compatible API.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports slow performance with Qwen3.8-Flash-Next on an M2 Max 96GB using the oMLX engine. Benchmarks give 124.1 pp TPS and 20.4 tg TPS at pp1024/tg128, 170.7 pp TPS and 16.7 tg TPS at pp4096/tg128, 214.3 pp TPS and 18.5 tg TPS at pp8192/tg128, and 235.7 pp TPS and 18.3 tg TPS at pp16384/tg128.
Setup is oMLX, a custom MLX-based engine, with 32768 context and a turboquant_kv_bits 8.0 KV cache. The user also tried oQ4e-fp16-mtp, oQ3-fp16-mtp and oQ3-MTP variants.
The fp16 variants are even slower at roughly 150 pp TPS and 8 tg TPS. The user notes llama.cpp achieves 350-400 pp TPS. The model is MoE with 6B active parameters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visioncoding
User reports a VLM is best overall for OCR and detail extraction, correctly reading mixed-script text (Chinese + Latin) and catching fine details other models missed.
Output is verbose at 1.4-2.2k tokens.
The user recommends it as the default for a coding-assistant MCP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingvision
User reports Qwen3.8 27B IQ4_XS with vision projector on an M2 MacBook Pro 32GB, at 21.9 t/s prompt and 8.6 t/s generation.
Setup is llama.cpp built from source, with context set to 128K due to memory constraints.
User notes the model is slow on Mac but that thinking quality is good, and mentions previous experience with Qwen3.5 35B-A3B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 22.38 t/s generation (tg128) and 360.22 t/s prompt processing (pp512) on a Mac Studio M2 Ultra with 192GB unified memory.
Setup is llama.cpp build b10450 llama-bench with the Unsloth UD-Q6_K_XL quant (24.13 GiB), flash attention on, q8_0 K and V cache and batch and micro-batch 2048.
The user serves the model through llama-server at 131072 context with MTP speculative decoding (draft-mtp, draft max 3), and gets around 16 t/s on the WebUI with a flappy bird game prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 25.8 t/s, with a 42 t/s peak, on an M2 Ultra with 60 cores and 192gb.
Setup is a custom llama.cpp fork with the model repacked losslessly to 141GiB, smaller than the public GGUFs, no KV cache quant, and an SSD KV cache with dynamic lanes, 1M context total across 8 lanes.
Prompt processing is about 250 t/s at 8k-32k. The user compares this with 16 t/s on an M3 Ultra.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User benchmarks Qwen3.8-27B on an Apple M2 Max 96GB, with MTPLX reaching 20-22 t/s decode and 109 t/s prefill.
Setup is 8-bit quants in GGUF and MLX with an unquantised KV cache at 262k context, across llama.cpp baseline, llama.cpp+MTP, MTPLX, oMLX, mlx-dspark (DFlash2/DSpark) and vllm-mlx, on an agentic coding benchmark with 4 phases.
MTPLX xhigh is worth the extra time, while oMLX xhigh is not. vllm-mlx leaks chain-of-thought into the output, and mlx-dspark DFlash2 xhigh thought 226K tokens and delivered nothing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports decode speeds at various context depths: 28 t/s at the start, 23.5 t/s at 45k, and 18 t/s at 192k.
The run was maintained with 8k token output.
Prefill performance is mentioned but no numbers are given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 31-35 t/s on an M5 MacBook Pro with a custom Swift/Metal engine.
Setup is an OpenAI-compatible server with streaming and tool-call support.
User reports a vision benchmark champion model at 70 s/img on an Apple M2 Max 96GB with llama.cpp b9690.
The model gives the best quality and stability with thinking disabled, and completed 90/90 successful runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User benchmarks Qwen 3.6 27B F16 on an M2 Max 96GB with llama.cpp and MTP speculative decoding, reporting 8-18 tok/s depending on the task.
Without MTP the same setup reaches 6.6 tok/s. A Q8 quant was also tested but performed worse.
Context up to 150k+ tokens is usable. The user notes chat template fixes were critical, and the setup was used for agentic coding to create a Pacman game.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports MTP speculative decoding gives a 2.5x speedup on an M2 Max 96GB.
Setup is Q5_K_M quant with a q4_0 KV cache.
The user also provides hardware recommendations for various Apple Silicon and NVIDIA GPUs.