Local model performance on your hardware

reported speed:

76.7 tokens/s generation · 2270.0 tokens/s prompt processing

quant:

oQ4 (MLX)

kv:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks oMLX 0.7.0 against 0.7.0rc1 on an M5 Ultra 256GB, running GLM-5.3-Flash oQ4 at 4K–200K context. Prefill improves from 792 to 2,270 tok/s (median) and decode from 55.4 to 76.7 tok/s. A 1M-token prompt completes with prefill 1,485 tok/s, decode 41.5 tok/s, and time to first token 11.8 min. Setup uses Lightning MTP, TurboQuant KV 4-bit, and one model loaded at a time.

reported speed:

68.8 tokens/s generation

quant:

oQ4e

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports GLM-5.3 Flash at 68.8 tok/s on an M5 Ultra 256GB. Setup is oMLX 0.7.0 with oQ4e quantization and MTP speculative decoding, with more optimization still to come. Prefill is 1,878 toks. User shared the result because other benchmarks looked lower than expected.

reported speed:

73.4 tokens/s generation · 1010.0 tokens/s prompt processing

quant:

Q3-G128-LSQ (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User asks whether oMLX benchmarks can be spoofed, citing a third-party oMLX benchmark entry for deepseek-v4.1-flash-q3-g128-lsq-mlx on an M5 Ultra with 256GB RAM, posted two days earlier, reporting about 73.4 tok/s generation and 1,010 tok/s prompt processing. The run is not the user's own; the figures come from a linked oMLX benchmark page and screenshot.

reported speed:

29.0 tokens/s generation

quant:

oQ8e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 8-bit at 29 t/s decode on a 1M-token prompt on an M5 Ultra 256GB Mac Studio. Setup is oMLX 0.7.0 with oQ8e weights and YaRN x4 position scaling; the 1M-token first read took 5.3 minutes and the next turn 6.7 s, finding all three hidden codes. At 250k context the model prefills at 4,235 t/s and decodes at 63 t/s; the user notes YaRN stretches a 262k-trained model and that a needle test does not prove reasoning at 1M.

reported speed:

59-74 tokens/s generation

quant:

oQ8e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares Qwen3.8-Flash-Next against Laguna-S-2.1 on a Mac Studio M5 Ultra 256GB, both capped at 262K context with thinking on and unique content per run. Qwen runs under oMLX with an oQ8e quant and MTP speculation, holding roughly 4,200 tok/s prefill and 59-74 tok/s decode across sizes. Laguna runs under LM Studio with an 8-bit quant, prefill degrading superlinearly from 10.3s at 8K to 455.4s at 200K, and decode falling from 68 to 34 tok/s with no speculation. Quality was a draw at 4/4 each on four script-verified problems. At 200K, prefill is 94% of total time on both. The user notes an earlier run was invalidated by shared prefixes letting the KV cache carry over.

reported speed:

108.0 tokens/s generation · 2887.0 tokens/s prompt processing

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 108 t/s generation and 2,887 t/s prompt processing on an M5 Ultra 256GB Mac Studio. Setup is oMLX 0.7.0.dev2 with MLX backend, oQ4e 4-bit quant, multi-token prediction depth 3, 16K prompt. M3 Ultra 512GB comparison: 70 t/s generation, 1,143 t/s prompt. At 256K context, M5 prompt processing was 2,544 t/s. Qwen3.8-27B on M5 Ultra: 48 t/s generation, 1,701 t/s prompt; RTX 5090 PC in LM Studio: 59 t/s generation, 3,031 t/s prompt.

reported speed:

50.0 tokens/s generation · 1800.0 tokens/s prompt processing

quant:

q4

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 50 t/s generation and 1800 t/s prompt processing on an M5 Ultra at 8k context. Figures come from the omlx website rather than a local run, with q4 weights and no MTP. The user notes the benchmarks' provenance is unclear but finds them reasonable. User calls the results very promising.