Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731-MXFP4-MLX decode at 14.89 tok/s on an M3 Ultra 512GB, about 2.8x below the 41.7 tok/s the model card reports for this checkpoint at 23K context with native DSpark MTP.
Setup is oMLX 0.6.4 with MXFP4 MLX weights, native MTP enabled with 3 draft tokens, 23,716-token prompt, no KV cache quant stated. MTP acceptance is 86-88% and prefill is 445.08 tok/s, but decode does not follow.
User notes the same ~14-15 tok/s class on 0.6.3rc3 and 0.6.4, no swap or memory pressure, and that 0.5.4rc2 (the reference build) was not A/B tested.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 4-bit at ~25 tok/s decode on an Apple M3 Ultra with 512 GB unified memory.
Setup is oMLX 0.6.3 serving the mlx-community 4-bit weights, the whole model in unified memory, non-streaming chat completions on localhost with max_tokens 400-512 and default sampling.
Speed was taken as completion tokens over wall time: two short-prompt runs at low reasoning effort gave 24.6 and 25.6 tok/s, one at xhigh effort 24.2 tok/s, and a ~1.5k-token prompt 17.9 tok/s including prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 109.7 tokens/s generation and 1770.1 tokens/s prefill on an Apple Mac Studio M3 Ultra with 96 GB unified memory.
Setup is MLX with 4-bit weights loaded via mlx_lm.load() and run through mlx_lm.generate.stream_generate() without an HTTP server, at a 512-token context setting, generating 50 tokens at temperature 0.0.
Source reports 20.20 GB peak memory and 77.4 MB KV memory for this row. 512 is the tested context setting, not a configured maximum; runtime version, GPU core count, KV-cache precision and maximum context are not stated. This is a short-context standalone result, not an end-to-end server throughput measurement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 107.5 t/s generation on an Apple Mac Studio with M3 Ultra and 256 GB unified memory.
Setup is mlx-lm 0.31.3 with the mlx-community 4-bit MLX quant, thinking off and no speculative decoding; peak memory was 5.24 GB.
The figure comes from a 34-token request that generated 1,156 tokens for a 900-word essay; the author ran two passes and reports variation of about 1 t/s or less. Configured context length is not stated for this run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-context
User benchmarks Qwen3.8-27B at 41.4 tok/s decode and 198.2 tok/s prefill at 1K context on an Apple M3 Max 128GB (40-core GPU).
Setup is MTPLX v2.8.2 with a 4-bit dynamic/GS32 MLX model, MTP depth 3, single request, 128-token generations, 3-minute GPU rests between rounds.
MTPLX decode falls to 19.8 tok/s and prefill to 115.5 tok/s at 16K, where oMLX oQ4e reaches 27.0 tok/s decode and 231.7 tok/s prefill; the 8-bit MTPLX Quality build is nearly identical in speed (18.8 tok/s, 121.3 tok/s) with KL 0.00105 versus Speed's 0.0220. ANE prefill on oMLX gains 57.0% at 32K (126.1 to 198.1 tok/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticsummarizationcreative-writinglong-context
User benchmarks Qwen3.5-35B-A3B (thinking disabled) on Apple Silicon, reporting effective throughput of 71.3 tok/s on the ops-agent scenario with an M3 Max 128GB (40 GPU) running oMLX at MLX 4-bit; generation speed was 90.8 tok/s.
Setup is oMLX with a tiered KV cache; the same M3 Max run gives 61.4 effective tok/s on doc-summary, 22.6 on prefill-test and 90.1 on creative-writing. Generation speed is similar across MLX engines (~55-93 tok/s depending on hardware) but prefill varies widely: at 8K context LM Studio MLX takes 49s to prefill while oMLX takes 1.7s with its persistent SSD cache from a prior run.
Effective tok/s is output tokens divided by total wall-clock time including prefill. On the same M3 Max, LM Studio MLX reaches 37.1 effective tok/s; on M1 Max 64GB (24 GPU) oMLX 4-bit reaches 37.5 and oMLX fp16 47.3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-use
User benchmarks Qwen3-8B at 25.2 tok/s on an M3 Pro 18GB.
Setup is llama.cpp b11246 with Q4_K_M and 1024-token input, 256 output tokens, single request.
Runs were taken under memory pressure so absolute speeds are pessimistic; MLX-LM reaches 35.5 tok/s at 4 concurrent requests while llama.cpp plateaus at 21.0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 25.67 t/s on an M3 Ultra 256GB.
Setup is Rapid-MLX 0.13.2 with mixed 4-bit affine MLX quantization and a 2048-token prefill chunk, at roughly 102.8-103.8GB active unified allocation.
Decode falls to 24.27 / 23.40 / 21.72 t/s at the 2048 / 8192 / 32768 prompt targets, with median TTFT of 2.262 / 9.236 / 44.659 seconds. Each row is the median of three batch-size-one runs with cold prefix cache and 256 requested decode tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 84.1 and 72.1 tok/s on production-shaped prompts and 87.0 tok/s at 12K context on a Mac Studio M3 Ultra 256GB.
Setup is mlx-serve 26.9.6-dev with a 4-bit MTP pack and native MTP; prefill measured at roughly 285-325 tok/s, and concurrency gave no gain (1.01x at N=4).
The same model class under Inco Splash reached 85.3-85.7 tok/s on short prompts and 65.7 tok/s at 32K, scaling 1.18x at N=2 and 1.27-1.29x at N=4 before going flat, with the engine reporting a maximum batch width of 4. The user notes the two engines' single-stream figures came from different prompts, so the 27B single-stream comparison is left undecided. Also measured: Qwen3.6-35B-A3B with mlx-dspark at accept length 4.5998 and 2.33x, versus MTPLX at accept 2.2294 and about 1.47x; Qwen3.8-Flash-Next with mlx-serve 26.9.6 and --mtp at 76.8 tok/s single stream (+32% over the same build without the flag) and 133.5 tok/s aggregate at N=8.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B at 98.8 tok/s decode and 2,358 tok/s prefill on an M3 Max 128GB using Splash 1.0.1 with 4-bit weights and DFlash speculative decoding.
Setup is Splash 1.0.1 on macOS 26.6.2, 4-bit proprietary format, ~22K context, everything resident on GPU. Warm TTFT at 22K is 98 ms; 4 parallel streams reach 181 tok/s aggregate.
User compares against Ollama NVFP4 (MLX) at 84.7 tok/s decode and oMLX serving Ornith 1.5 at 96.2 tok/s, and notes DFlash gains 2.74x on predictable text (239 tok/s) versus 87.2 tok/s on technical prose. User also benchmarks Qwen3.8-27B dense at 30.0 tok/s decode and 292 tok/s prefill, below Ollama's MLX engine at 42.5 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks four MLX engines on a 14" M3 Max MacBook Pro (36 GB) across Qwen3.6-35B-A3B and Qwen3.8-27B at 4-bit.
Rapid-MLX leads the MoE at 110.2 tok/s decode, while Splash leads the dense 27B at 31.7 tok/s (MTPLX 31.5, effectively a tie). oMLX trails on dense at 17.9 tok/s but is level with the leaders on the MoE at 104.2 tok/s.
Each engine used its own weight build: Rapid and MTPLX ran MTP-augmented 4-bit builds, Splash paired its own DFlash2 draft, and oMLX ran the plain mlx-community 4-bit. Sustained test was 3 runs x 256 output tokens, median, temperature 0, thinking off. User kept Rapid-MLX because the MoE is their daily model. Qwen3.8-Flash-Next (125B) does not fit on 36 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Minimax-m2.1-4bit at 25 t/s generation on an M3 Ultra 256GB with 80-core GPU, compared against llama.cpp with flash attention at 32 t/s.
Setup is MLX with 4bit weights at 30K context; the same model under llama.cpp with flash attention reached 32 t/s generation and 78s prompt processing.
At 146K context MLX fell to 5.95 t/s generation versus 12.12 t/s for llama.cpp, and the user says MLX runs about 50% slower on long-context generation, impacting agentic coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 4-bit served across two Mac Studio M3 Ultra (512GB) boxes: prefill 1012 tok/s on a 13.9K-token prompt (TTFT 13.7 s), decode 45-47 tok/s single-stream and 112-117 tok/s aggregate at batch 8.
Setup is MLX with an oMLX 0.6.3rc2 serving overlay, MTP/speculative machinery enabled, over a single Thunderbolt 5 cable using the jaccl RDMA mesh backend.
The decode figure is given only as a range, so no single generationTps is recorded. The repo is a reproducibility playbook; the oMLX 0.6.3rc2 build artifact is no longer available and must be obtained upstream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash Next Q4 running through ds4 on an M3 Ultra 96GB with the full 262K context configured, delivering 55–60 tok/s decode and 667 tok/s prefill.
About 80GB is used for model, KV cache and buffers, while the model's 95GB n-gram table streams from SSD.
The user runs it daily through a codex harness as a headless box.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingcreative-writing
User compares Inco Splash 1.1 and oMLX 0.7.0rc1 running an abliterated Qwen3.8-27B on an M3 Max 48GB MacBook Pro. Splash with the DFlash2 draft model reached 91.8 tok/s on code prompts, 52.6 tok/s on creative writing, and 37.7 tok/s on stock Qwen, while oMLX with MTP was in the low-to-mid 30s tok/s. Setup used MLX 4-bit weights on Splash and oQ4e on oMLX, thinking off, temp 0.7, max 600 tokens, median of 3 runs, one engine at a time. User notes the quant is not identical between engines, prompts were short, and long context and concurrent requests were not tested.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash 4bit at 29.9 tok/s generation and 426.5 tok/s prompt processing on a Mac Studio with M3 Ultra 512GB unified memory.
Setup is MLX 0.32.0 (draft PR #1189) with a standard 8,000-byte prompt, 300 generated tokens, temperature 0, and no speculative decoding.
The same machine also ran DeepSeek R1-0528 4bit MLX at 206.9 / 20.3, GLM-5.2 4bit MLX at 182.6 / 17.9, Kimi K2.7 Code Q2_K_XL GGUF on llama.cpp 571d0d5 at 144.8 / 24.6, Qwen3.5 397B-A17B 4bit MLX at 510.3 / 38.1, gpt-oss-120B MXFP4 MLX at 1,405.9 / 79.2, Qwen3.6 35B-A3B 4bit MLX at 2,892.2 / 94.6, Gemma 4 31B IT 4bit at 347.8 / 29.2, Gemma 4 26B-A4B IT 4bit at 2,508.5 / 104.9, Qwen3-Coder-Next 4bit at 2,099.8 / 77.2, and Devstral 2 123B 4bit at 90.4 / 8.9 (prompt / generation tok/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 25.7 t/s aggregate decode on an M3 Ultra 512GB, with throughput collapsing as concurrent long sequences are batched.
Setup is llama.cpp (Unsloth prebuilt b10830) with Q8_0 GGUF, 32k context, flash attention on, and 8 slots. At 32k context, N=1 gives 25.7 t/s aggregate, N=2 gives 8.6 t/s, and N=4 gives 5.5 t/s. Two separate -np 1 processes reach 39.4 t/s aggregate, 4.6x faster than one process batching the same two sequences.
User also sees the same cliff with GLM-5.3-Flash, going from 40 to 10.6 t/s aggregate at N=4 between 256 and 8k context, and uses two -np 1 processes behind a load balancer as a production workaround.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
multilingualcodingvisionlong-context
User reports Kimi K3 REAP73 at 5.51 t/s on a 512 GiB M3 Ultra.
Setup is mlx-lm with mxfp4 experts and Q8 non-experts, 242 of 896 experts retained, single machine.
Figures are the mean of three prompts x 96 tokens; the 350 GB REAP80 build decodes at 5.54 t/s and the graded two-bank build at 2.68 t/s. Prefill runs at ~73 tok/s. Unpruned tiers are 883 GB and above and do not fit any Apple Silicon machine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Kimi-K3 (2.78T, MXFP4) at ~2.0 tok/s single-stream decode on four M3 Ultra 512GB Mac Studios in tensor parallel over a Thunderbolt 5 RDMA mesh.
Setup is MLX 0.32.0 / mlx-lm 0.31.3 with MXFP4 weights, full 1,048,576-token context (29 GB KV), 420.8 GB resident per rank, 229.6 s load. Aggregate throughput reaches 21.2 tok/s at 32 concurrent streams; chunked prefill is ~170 tok/s.
User attributes decode at batch 1 to fixed per-call dispatch overhead (~170 ms/token) rather than bandwidth, with collectives at 1% and MoE gather at 7%; ~330 ms of the ~500 ms/token remains unattributed. An unresolved multi-chunk prefill hang under TP is described, following the host rather than the rank index.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8-27B at about 65-75 tok/s generation at 8K context on one Mac Studio M3 Ultra 512GB (one of four run as independent replicas).
Setup is omlx 0.6.1 with an oQ4e MLX quant, MTP speculative decoding with 3 draft tokens, and a 262144-token context cap. Prefill measured at 420-442 tok/s; batch-8 throughput reached 274 tok/s.
MTP draft depth 3 was the peak (65-75 tok/s vs 50.0 at depth 1 and 65.6 at depth 4). oQ8e was tied on quality but slower at 43 tok/s. A Mac mini M4 Pro 64GB reached 20.8 tok/s and OOM'd around 168k context. An RTX 5090 reference run under llama.cpp with NVFP4+MTP hit ~145 tok/s.