Local model performance on your hardware

generation:

14.9 tokens/s

prompt processing (prefill):

445.1 tokens/s

quant:

MXFP4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731-MXFP4-MLX decode at 14.89 tok/s on an M3 Ultra 512GB, about 2.8x below the 41.7 tok/s the model card reports for this checkpoint at 23K context with native DSpark MTP. Setup is oMLX 0.6.4 with MXFP4 MLX weights, native MTP enabled with 3 draft tokens, 23,716-token prompt, no KV cache quant stated. MTP acceptance is 86-88% and prefill is 445.08 tok/s, but decode does not follow. User notes the same ~14-15 tok/s class on 0.6.3rc3 and 0.6.4, no swap or memory pressure, and that 0.5.4rc2 (the reference build) was not A/B tested.

generation:

25.0 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 4-bit at ~25 tok/s decode on an Apple M3 Ultra with 512 GB unified memory. Setup is oMLX 0.6.3 serving the mlx-community 4-bit weights, the whole model in unified memory, non-streaming chat completions on localhost with max_tokens 400-512 and default sampling. Speed was taken as completion tokens over wall time: two short-prompt runs at low reasoning effort gave 24.6 and 25.6 tok/s, one at xhigh effort 24.2 tok/s, and a ~1.5k-token prompt 17.9 tok/s including prefill.

generation:

65-75 tokens/s

prompt processing (prefill):

420-442 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 65-75 tok/s generation at 8K context on one Mac Studio M3 Ultra 512GB (one of four run as independent replicas). Setup is omlx 0.6.1 with an oQ4e MLX quant, MTP speculative decoding with 3 draft tokens, and a 262144-token context cap. Prefill measured at 420-442 tok/s; batch-8 throughput reached 274 tok/s. MTP draft depth 3 was the peak (65-75 tok/s vs 50.0 at depth 1 and 65.6 at depth 4). oQ8e was tied on quality but slower at 43 tok/s. A Mac mini M4 Pro 64GB reached 20.8 tok/s and OOM'd around 168k context. An RTX 5090 reference run under llama.cpp with NVFP4+MTP hit ~145 tok/s.

generation:

4-7 tokens/s

quant:

MXFP4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running DeepSeek V4 Flash 0731 on a Mac Studio M3 Ultra with 512GB unified memory, served by oMLX with the MXFP4 MLX weights. Setup uses oMLX with the Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX model and Hermes Agent pointed at the oMLX endpoint. An earlier Unsloth GGUF attempt ran at around 4-7 tok/s because it fell back to CPU inference. The user hit a cache invalidation problem after 130K tokens, caused by the hot cache size being capped at 30GB, and reports tool-call failures in Unsloth Studio that were traced to the UI not sending the tool template.

generation:

19.7 tokens/s

prompt processing (prefill):

439.4 tokens/s

quant:

oQ4e

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash on an M3 Ultra 512GB with oQ4e and Engram in RAM, reaching 439.4 t/s prefill and 19.7 t/s generation at 64K context with MTP off, and 435.6 t/s prefill and 39.7 t/s generation with MTP on. Setup is oMLX 0.7.0.dev2 with Python code prompts, temperature 1.0, 128 generated tokens, and no prefix cache. One measured run per configuration after warm-up. At 4K context, prefill was 458.0 t/s with MTP off and 452.2 t/s with MTP on, while generation was 20.2 t/s and 32.1 t/s respectively. At 16K, prefill was 459.1 t/s and 454.8 t/s, with generation at 20.0 t/s and 34.7 t/s. At 32K, prefill was 452.2 t/s and 447.7 t/s, with generation at 19.8 t/s and 31.5 t/s. The user also notes experimental MoE Expert SSD Offload support for DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE, and an M5 prefill speedup from 615.2 to 826.7 t/s at 32K on an M5 Max with Qwen3.8-27B using INT8-activation kernels.

generation:

20.0 tokens/s

prompt processing (prefill):

533.0 tokens/s

quant:

8bit affine (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports an optimized DeepSeek V4 Flash 8-bit affine MLX model on oMLX, with prefill improving from ~300-321 to ~533 tok/s and decode from ~7.31 to ~20-22 tok/s. Real runs at 79K-119K context show 19.2-20.7 tok/s. User asks for community review on accuracy and next optimization directions.