Local model performance on your hardware

generation:

198.6 tokens/s

prompt processing (prefill):

3657.8 tokens/s

quant:

oQ5e

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B at 198.6 tok/s generation and 3657.8 tok/s prompt processing on an M5 Max 128GB. Setup is oMLX 0.7.1dev2 with oQ5e quant, Lightning MTP speculative decoding, and TurboQuant off, at 1024-token prompt and 128-token generation. Splash engine reaches 182.0 tok/s generation and 6040.8 tok/s prompt processing on the same test; continuous batching at 4x reaches 275.2 tok/s aggregate generation. A second model, Qwen3.7-27B, is also benchmarked.

generation:

8.4 tokens/s

quant:

oQ4e

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 oQ4e-mtp restored-prefix B3 / PR1 61bd8a06 at 8.4 t/s on an M5 Max 128GB. Setup is oMLX with oQ4e 4-bit weights and TurboQuant 4-bit KV cache, three greedy requests batched at roughly 20k / 45k / 135k prompt tokens with SSD prefix cache enabled. The 8.4 t/s is the per-row rate at the 20k row, measured at 8.35 t/s; the other rows ran 8.83 and 7.84 t/s per row, and the batch phase took 66 seconds.

generation:

36.1 tokens/s

prompt processing (prefill):

498.3 tokens/s

quant:

2.44bit-mixed (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 36.1 t/s generation and 498.3 t/s prefill on a MacBook Pro M5 Max 128GB 40 GPU. Setup is oMLX 0.5.7 with a 2.44 bpw mixed MLX quantization (92.8 GB on disk) at 1k prompt, single request, 128 generated tokens, speculative decoding off. Generation holds 33.8 t/s at 4k and 31.5 t/s at 32k, with peak memory rising 1.03 GB across an 8x context jump; prefill at 32k takes 97 seconds to first token. Continuous batching reaches 83.2 t/s at batch 8 (2.30x).