Local model performance on your hardware

generation:

121.2 tokens/s

quant:

4bit (MLX)

User benchmarks dflash-mlx at 121.2 t/s decode on a 32,768-token prompt, the fastest of four MLX frameworks tested on an Apple M5 Max with 64 GB unified memory. Setup is dflash-mlx v0.1.7 with the 4-bit MLX Qwen3.6-35B-A3B model and a companion draft model for speculative decoding, single sequence, prefix caching disabled, cooldown=60 between runs. Decode medians across the ladder: 148.7 t/s at 64 tokens, 138.9 at 512, 154.5 at 2,048, 127.7 at 4,096, 124.6 at 8,192, 119.0 at 16,384, and 121.2 at 32,768. TTFT at 32K is 8.1 s. The other three frameworks were tested at cooldown=2 and would score higher at the longer cooldown.

generation:

34.6 tokens/s

prompt processing (prefill):

400.0 tokens/s

quant:

oQ8e

User reports Swift-1.5-Qwen3.8-27b-oQ8e-mtp at 34.6 tok/s generation and around 400 tok/s prompt processing on an Apple M5 Max 64GB. Setup is oMLX with oQ8e quant, 262k context window, thinking on at xhigh, MTP speculative decoding enabled. User compares against base Qwen3.8 27B (32.8 tok/s, ~400 tok/s prompt processing) and notes Swift 1.5 generates fewer tokens per run (51k vs 77k average) with a similar quality score (85.8 vs 85.0).

generation:

30.0 tokens/s

quant:

Q2-Q4 mixed imatrix (GGUF)

User reports about 30 t/s on an M5 Max with a quantized DeepSeek V4 Flash running on the DS4 engine. The user states this is double the speed of llama.cpp. The user asks for feedback from CUDA and ROCm users.

generation:

97.0 tokens/s

User reports Gemma 4 26B at 97 tokens/s in llama.cpp on a MacBook Pro M5 Max. Setup is GGUF weights, with a prompt asking for a recursive Python Fibonacci program. With the user's Multi-Token Prediction (MTP) implementation for llama.cpp and quantized Gemma 4 assistant draft models, the same prompt runs at 138 tokens/s, which the user calls 40% faster.

generation:

63.0 tokens/s

quant:

4-bit (MLX)

User reports 63 t/s on Qwen3.6-27B 4-bit MLX on an M5 Max 64GB with the MTPLX engine, up from a 28 t/s baseline. Setup is a custom patched MLX fork with Metal kernels, using native MTP heads at temperature 0.6, top_p 0.95 and top_k 20, with optimal depth D3.

generation:

32.0 tokens/s

User reports Qwen 3.6 27B at 32 t/s on a MacBook Pro M5 Max 64GB, generating 33,946 tokens in 18m04s. User compares Gemma 4 31B on the same hardware at 27 t/s, generating 6,209 tokens in 3m51s. User finds Qwen showed more creativity while Gemma won for game logic.