MLX affine 4-bit, group 64, 4.501 effective bpw (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 37.9 t/s on an M3 Ultra 512GB, from a single run of one 68-token prompt and 120 generated tokens.
Setup is MLX with affine 4-bit quantization, group size 64 and 4.501 effective bpw, peaking at 15.5 GB of memory.
The author describes the figure as a rough guide rather than a repeated controlled benchmark; no prompt-processing throughput or TTFT is given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
vision
User reports Qwen3.8 at 13.2 t/s on an M3 Ultra with 512GB unified memory.
Setup is mlx-vlm 0.6.13 with the unquantized bf16 MLX VLM build, peak memory 55.8 GB.
The figure comes from one 470-token image-plus-text prompt, one run per build, presented as an order-of-magnitude guide rather than repeated benchmarking.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Tiel-Coder 35B-A3B at 201.3 t/s median native decode on an Apple Mac Studio M3 Ultra with 512 GiB unified memory.
Setup is AX Engine with the AXQ-MXFP4 MLX export, explicit throughput-MTP at maximum draft depth three, expert streaming off, and cold request KV with no prefix reuse.
The figure is native decode only on a python-lru prompt with 194 input tokens and 256 total output tokens; completion throughput including TTFT is 160.35 t/s and median native TTFT is 329.74 ms.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Llama 3.1 405B at 2.99 t/s on a single M3 Ultra 512GB node.
Setup is MLX with Q4 quantisation at 1K context, single node.
This is part of a 200+ configuration campaign across 5 models and 9 topologies on a 5-node M3 Ultra cluster; the report also covers Qwen 2.5 32B, Mixtral 8x7B, DeepSeek V3.2 and Kimi K2.5, with TP and PP scaling results.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731-MXFP4-MLX decode at 14.89 tok/s on an M3 Ultra 512GB, about 2.8x below the 41.7 tok/s the model card reports for this checkpoint at 23K context with native DSpark MTP.
Setup is oMLX 0.6.4 with MXFP4 MLX weights, native MTP enabled with 3 draft tokens, 23,716-token prompt, no KV cache quant stated. MTP acceptance is 86-88% and prefill is 445.08 tok/s, but decode does not follow.
User notes the same ~14-15 tok/s class on 0.6.3rc3 and 0.6.4, no swap or memory pressure, and that 0.5.4rc2 (the reference build) was not A/B tested.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 4-bit at ~25 tok/s decode on an Apple M3 Ultra with 512 GB unified memory.
Setup is oMLX 0.6.3 serving the mlx-community 4-bit weights, the whole model in unified memory, non-streaming chat completions on localhost with max_tokens 400-512 and default sampling.
Speed was taken as completion tokens over wall time: two short-prompt runs at low reasoning effort gave 24.6 and 25.6 tok/s, one at xhigh effort 24.2 tok/s, and a ~1.5k-token prompt 17.9 tok/s including prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 4-bit served across two Mac Studio M3 Ultra (512GB) boxes: prefill 1012 tok/s on a 13.9K-token prompt (TTFT 13.7 s), decode 45-47 tok/s single-stream and 112-117 tok/s aggregate at batch 8.
Setup is MLX with an oMLX 0.6.3rc2 serving overlay, MTP/speculative machinery enabled, over a single Thunderbolt 5 cable using the jaccl RDMA mesh backend.
The decode figure is given only as a range, so no single generationTps is recorded. The repo is a reproducibility playbook; the oMLX 0.6.3rc2 build artifact is no longer available and must be obtained upstream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash 4bit at 29.9 tok/s generation and 426.5 tok/s prompt processing on a Mac Studio with M3 Ultra 512GB unified memory.
Setup is MLX 0.32.0 (draft PR #1189) with a standard 8,000-byte prompt, 300 generated tokens, temperature 0, and no speculative decoding.
The same machine also ran DeepSeek R1-0528 4bit MLX at 206.9 / 20.3, GLM-5.2 4bit MLX at 182.6 / 17.9, Kimi K2.7 Code Q2_K_XL GGUF on llama.cpp 571d0d5 at 144.8 / 24.6, Qwen3.5 397B-A17B 4bit MLX at 510.3 / 38.1, gpt-oss-120B MXFP4 MLX at 1,405.9 / 79.2, Qwen3.6 35B-A3B 4bit MLX at 2,892.2 / 94.6, Gemma 4 31B IT 4bit at 347.8 / 29.2, Gemma 4 26B-A4B IT 4bit at 2,508.5 / 104.9, Qwen3-Coder-Next 4bit at 2,099.8 / 77.2, and Devstral 2 123B 4bit at 90.4 / 8.9 (prompt / generation tok/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 25.7 t/s aggregate decode on an M3 Ultra 512GB, with throughput collapsing as concurrent long sequences are batched.
Setup is llama.cpp (Unsloth prebuilt b10830) with Q8_0 GGUF, 32k context, flash attention on, and 8 slots. At 32k context, N=1 gives 25.7 t/s aggregate, N=2 gives 8.6 t/s, and N=4 gives 5.5 t/s. Two separate -np 1 processes reach 39.4 t/s aggregate, 4.6x faster than one process batching the same two sequences.
User also sees the same cliff with GLM-5.3-Flash, going from 40 to 10.6 t/s aggregate at N=4 between 256 and 8k context, and uses two -np 1 processes behind a load balancer as a production workaround.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
multilingualcodingvisionlong-context
User reports Kimi K3 REAP73 at 5.51 t/s on a 512 GiB M3 Ultra.
Setup is mlx-lm with mxfp4 experts and Q8 non-experts, 242 of 896 experts retained, single machine.
Figures are the mean of three prompts x 96 tokens; the 350 GB REAP80 build decodes at 5.54 t/s and the graded two-bank build at 2.68 t/s. Prefill runs at ~73 tok/s. Unpruned tiers are 883 GB and above and do not fit any Apple Silicon machine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Kimi-K3 (2.78T, MXFP4) at ~2.0 tok/s single-stream decode on four M3 Ultra 512GB Mac Studios in tensor parallel over a Thunderbolt 5 RDMA mesh.
Setup is MLX 0.32.0 / mlx-lm 0.31.3 with MXFP4 weights, full 1,048,576-token context (29 GB KV), 420.8 GB resident per rank, 229.6 s load. Aggregate throughput reaches 21.2 tok/s at 32 concurrent streams; chunked prefill is ~170 tok/s.
User attributes decode at batch 1 to fixed per-call dispatch overhead (~170 ms/token) rather than bandwidth, with collectives at 1% and MoE gather at 7%; ~330 ms of the ~500 ms/token remains unattributed. An unresolved multi-chunk prefill hang under TP is described, following the host rather than the rank index.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8-27B at about 65-75 tok/s generation at 8K context on one Mac Studio M3 Ultra 512GB (one of four run as independent replicas).
Setup is omlx 0.6.1 with an oQ4e MLX quant, MTP speculative decoding with 3 draft tokens, and a 262144-token context cap. Prefill measured at 420-442 tok/s; batch-8 throughput reached 274 tok/s.
MTP draft depth 3 was the peak (65-75 tok/s vs 50.0 at depth 1 and 65.6 at depth 4). oQ8e was tied on quality but slower at 43 tok/s. A Mac mini M4 Pro 64GB reached 20.8 tok/s and OOM'd around 168k context. An RTX 5090 reference run under llama.cpp with NVFP4+MTP hit ~145 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 32B at 31.2 t/s generation on 1K context using an M3 Ultra 512GB.
Setup is MLX with Q4 quantization and FP16 KV cache, batch size 1, 256 generation tokens, median of 3 trials.
The post reports a systematic sweep of 5 models, 6 quants, and 7 context lengths (276 runs total), showing generation drops to 8.5 t/s at 128K context and prompt processing falls from ~345 t/s at 1K to ~154 t/s at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagenticlong-context
User reports running DeepSeek V4 Flash 0731 on a Mac Studio M3 Ultra with 512GB unified memory, served by oMLX with the MXFP4 MLX weights.
Setup uses oMLX with the Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX model and Hermes Agent pointed at the oMLX endpoint. An earlier Unsloth GGUF attempt ran at around 4-7 tok/s because it fell back to CPU inference.
The user hit a cache invalidation problem after 130K tokens, caused by the hot cache size being capped at 30GB, and reports tool-call failures in Unsloth Studio that were traced to the UI not sending the tool template.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-usecoding
User reports DeepSeek V4.1 Flash at 300 t/s prefill and 16 t/s decode on an M3 Ultra 512GB, running the Q2 quant on the DS4 engine.
Tool calls were functional but the model made questionable choices, ignoring the code execution tool and searching the local drive for a remote file before finding the right path. The 18k-token Hermes prompt was insufficient to steer it.
At around 110k tokens of context the decode rate only just began to dip below 16 t/s, which the user notes is a key advantage of DS4 over llama.cpp. The GPU ran at about 95% throughout. The user compares it unfavorably to GLM 5.3 Flash, which puts out roughly 21-19 t/s, and is hesitant to try Q4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 31.05 t/s generation and 558.83 t/s prompt processing on an M3 Ultra 512GB.
Setup is llama.cpp through llama-swap with the unsloth Q4_K_XL GGUF at 256,000 context, tested with llama-benchy.
Prompt processing measured at pp1000 @ d1000 with 3312 ms time to first response; generation measured at tg500 @ d1000 with a peak of 31.67 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports DeepSeek V4.1 Flash at 31.3 t/s decode at 8k context on an M3 Ultra 512GB, up from 16.6 t/s upstream.
Setup is a forked ds4 engine with Q4 GGUF weights, DSpark multi-token speculative decoding, and a disk KV cache. At 300k context decode is 28.3 t/s, prefill on a 62k prompt is 813 t/s, and TTFT on a 23k system prompt is 31.1 s.
With DSpark on, code generation reaches 40.5 t/s and agent answer phases 41.3 t/s. A 91-minute agent turn decoded 101k tokens with 4.6M tokens prefilled at 99.5% cache hit and 56 tool calls. Output is byte-identical to upstream under greedy decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports DeepSeek V4.1 Flash on an M3 Ultra 512GB with oQ4e and Engram in RAM, reaching 439.4 t/s prefill and 19.7 t/s generation at 64K context with MTP off, and 435.6 t/s prefill and 39.7 t/s generation with MTP on.
Setup is oMLX 0.7.0.dev2 with Python code prompts, temperature 1.0, 128 generated tokens, and no prefix cache. One measured run per configuration after warm-up.
At 4K context, prefill was 458.0 t/s with MTP off and 452.2 t/s with MTP on, while generation was 20.2 t/s and 32.1 t/s respectively. At 16K, prefill was 459.1 t/s and 454.8 t/s, with generation at 20.0 t/s and 34.7 t/s. At 32K, prefill was 452.2 t/s and 447.7 t/s, with generation at 19.8 t/s and 31.5 t/s. The user also notes experimental MoE Expert SSD Offload support for DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE, and an M5 prefill speedup from 615.2 to 826.7 t/s at 32K on an M5 Max with Qwen3.8-27B using INT8-activation kernels.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports an optimized DeepSeek V4 Flash 8-bit affine MLX model on oMLX, with prefill improving from ~300-321 to ~533 tok/s and decode from ~7.31 to ~20-22 tok/s.
Real runs at 79K-119K context show 19.2-20.7 tok/s.
User asks for community review on accuracy and next optimization directions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8 t/s on a Mac Studio M3 Ultra 512GB with DeepSeek V4 Flash GGUF Q4_K_XL.
The user also tried Q8.
The user expected better performance.