Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-uselong-context
User reports DeepSeek V4 Flash at ~22 tok/s decode and ~75 tok/s prefill on a Mac Studio M3 Ultra with 256GB unified memory.
Setup is vMLX with a mixed-precision MLX affine quantization (Q4 routed experts, Q8 attention and shared expert, fp16 norms) at group size 32, 173 GB on disk, 1,048,576 token context.
vMLX disables continuous batching for this model, so all requests go through SimpleEngine; the bundle requires a runtime patch to vMLX's jang_tools loader to avoid 8-bit expert wrapping and BOS-token loops.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports a Qwen3.8-Flash-Next MTP draft head for GGUFs on llama.cpp, measured on an M3 Max 128GB with Metal.
Setup is llama.cpp with an IQ3_XXS trunk and an unsloth shared Q8_0 MTP head, one session.
Decode is 31.7-35.2 tok/s with MTP against 23.8-27.4 without, draft acceptance 84-99% at 2 drafts. Five concurrent agent sessions cost 8-20% throughput at 2 drafts, while 1 draft at 89% acceptance is neutral. The work is blocked on unmerged llama.cpp PR #28243.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 32B at 31.2 t/s generation on 1K context using an M3 Ultra 512GB.
Setup is MLX with Q4 quantization and FP16 KV cache, batch size 1, 256 generation tokens, median of 3 trials.
The post reports a systematic sweep of 5 models, 6 quants, and 7 context lengths (276 runs total), showing generation drops to 8.5 t/s at 128K context and prompt processing falls from ~345 t/s at 1K to ~154 t/s at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagenticlong-context
User reports running DeepSeek V4 Flash 0731 on a Mac Studio M3 Ultra with 512GB unified memory, served by oMLX with the MXFP4 MLX weights.
Setup uses oMLX with the Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX model and Hermes Agent pointed at the oMLX endpoint. An earlier Unsloth GGUF attempt ran at around 4-7 tok/s because it fell back to CPU inference.
The user hit a cache invalidation problem after 130K tokens, caused by the hot cache size being capped at 30GB, and reports tool-call failures in Unsloth Studio that were traced to the UI not sending the tool template.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-usecoding
User reports DeepSeek V4.1 Flash at 300 t/s prefill and 16 t/s decode on an M3 Ultra 512GB, running the Q2 quant on the DS4 engine.
Tool calls were functional but the model made questionable choices, ignoring the code execution tool and searching the local drive for a remote file before finding the right path. The 18k-token Hermes prompt was insufficient to steer it.
At around 110k tokens of context the decode rate only just began to dip below 16 t/s, which the user notes is a key advantage of DS4 over llama.cpp. The GPU ran at about 95% throughout. The user compares it unfavorably to GLM 5.3 Flash, which puts out roughly 21-19 t/s, and is hesitant to try Q4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 31.05 t/s generation and 558.83 t/s prompt processing on an M3 Ultra 512GB.
Setup is llama.cpp through llama-swap with the unsloth Q4_K_XL GGUF at 256,000 context, tested with llama-benchy.
Prompt processing measured at pp1000 @ d1000 with 3312 ms time to first response; generation measured at tg500 @ d1000 with a peak of 31.67 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports DeepSeek V4.1 Flash at 31.3 t/s decode at 8k context on an M3 Ultra 512GB, up from 16.6 t/s upstream.
Setup is a forked ds4 engine with Q4 GGUF weights, DSpark multi-token speculative decoding, and a disk KV cache. At 300k context decode is 28.3 t/s, prefill on a 62k prompt is 813 t/s, and TTFT on a 23k system prompt is 31.1 s.
With DSpark on, code generation reaches 40.5 t/s and agent answer phases 41.3 t/s. A 91-minute agent turn decoded 101k tokens with 4.6M tokens prefilled at 99.5% cache hit and 56 tool calls. Output is byte-identical to upstream under greedy decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 27B at 7.4 t/s generation and 121.3 t/s prompt processing on an M3 Max 96GB.
Setup is oMLX with the MLX 8-bit model at pp1024/tg128, using 28.34 GB peak memory.
A pp4096/tg128 run reached 8.8 t/s generation and 133.8 t/s prompt processing. Continuous batching at 4x reached 19.9 t/s aggregate generation. User is new to LLMs and asks whether the slow speed is due to the dense model or a setup problem.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports DeepSeek V4.1 Flash on an M3 Ultra 512GB with oQ4e and Engram in RAM, reaching 439.4 t/s prefill and 19.7 t/s generation at 64K context with MTP off, and 435.6 t/s prefill and 39.7 t/s generation with MTP on.
Setup is oMLX 0.7.0.dev2 with Python code prompts, temperature 1.0, 128 generated tokens, and no prefix cache. One measured run per configuration after warm-up.
At 4K context, prefill was 458.0 t/s with MTP off and 452.2 t/s with MTP on, while generation was 20.2 t/s and 32.1 t/s respectively. At 16K, prefill was 459.1 t/s and 454.8 t/s, with generation at 20.0 t/s and 34.7 t/s. At 32K, prefill was 452.2 t/s and 447.7 t/s, with generation at 19.8 t/s and 31.5 t/s. The user also notes experimental MoE Expert SSD Offload support for DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE, and an M5 prefill speedup from 615.2 to 826.7 t/s at 32K on an M5 Max with Qwen3.8-27B using INT8-activation kernels.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a bug in oMLX 0.6.4 distributed clustering where the coordinator fails to release RAM and GPU after a crash. The model is mlx-community/MiniMax-M3-4bit at 236 GB, run across rank 0 on a Mac Studio M3 Ultra 256GB and rank 1 on a Mac Studio M2 Ultra 192GB.
The first completion produced 17 tokens from a 7,693-token prompt at 17.3 tok/s. The crash was triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching.
After the crash, roughly 116 GB of wired memory has no owning process and the GPU stays pinned at 100%, requiring a reboot. The M2 Ultra worker released memory cleanly. The user also reports that MiniMax-M3 crashes the cluster after the first prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports Qwen3.8-Flash-Next at ~38 tok/s decode on a 48 GB M3 Max, with prefill at 210-380 tok/s.
Setup is the user's oMLX fork with a custom Metal kernel for ternary routed expert gate/up (Bonsai T5 packing, ~1.875 bpw), Q3 expert down projections and the 53 GB n-gram table left on SSD as Q8 and mmapped per token, about 35.6 GiB resident. The oMLX memory guard is on the 'safe' profile with a 48 GB limit, one model loaded and one request at a time.
A 65,536-token prompt plus 256 output completed at 30.7 tok/s with a 42.3 GiB physical peak, and 96K trips the prefill guard. Against Unsloth UD-Q4_K_XL, which does not fit in 48 GB, MMLU is 83.0% vs 89.7% and KLD vs Q8_0 is 0.49 vs 0.036.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodingtool-use
User reports GLM-5.3-Flash Q4 at 37.4 t/s end to end at 300k context on an M3 Ultra, against 21.6 t/s for stock ds4.
Setup is a ds4 branch tuned for the M3 Ultra, decoding serially in a single stream without the dflash drafter, with prefill at 550 t/s at 62k.
Serial decode rises from 29 to 40 t/s at short context and from 24 to 38 at 62k. In a claude code harness at ~200k depth it averages over 38 t/s output and reaches 60 t/s writing SQL, where the optional dflash drafter adds +20 to +50 percent.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 11-13 t/s on an M3 Max 96GB with SSD streaming and iogpu.wired_limit_mb=86016.
Setup is antirez's ds4 engine with GGUF. TTFT is 3-5s after warmup.
Prefill of 36k tokens takes about 2.5 minutes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports an optimized DeepSeek V4 Flash 8-bit affine MLX model on oMLX, with prefill improving from ~300-321 to ~533 tok/s and decode from ~7.31 to ~20-22 tok/s.
Real runs at 79K-119K context show 19.2-20.7 tok/s.
User asks for community review on accuracy and next optimization directions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8 t/s on a Mac Studio M3 Ultra 512GB with DeepSeek V4 Flash GGUF Q4_K_XL.
The user also tried Q8.
The user expected better performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash serving on an M3 Ultra 512GB with the ds4 engine, with cold prefill improved from 392 to 475 t/s at 64k context through kernel optimizations.
Cache prewarming with max_tokens:0 yields roughly 10x speedup for chat turns, from 6-20s down to 1.6s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarization
User reports a custom C + Metal runtime with speculative decoding at 18 t/s end-to-end on coding tasks and 17.7 t/s for an LRUCache implementation, on 36 GB unified memory.
Q4 weights are about 15 GB, mmap'd. Prose runs at roughly 10-11 t/s. TTFT is about 1.4s for a short prompt and about 2.7s for a 128-token prompt.
The user compares the custom runtime against llama.cpp and reports it faster.
User compares Mac Studio configs (M5 Ultra 96GB, M5 Max 128GB, M5 Max 64GB) for local LLM use.
They discuss Qwen3.8-27B at Q8 and Qwen3.8-Flash-Next (6B active) at various quants, including Flash-Next on 96GB Ultra with UD-IQ4_XS (93.7GB, 91.1% retention) and on 128GB Max with UD-Q4_K_XL (111GB, 93.5% retention), with the N-gram layer offloaded to SSD.
User asks for advice on bandwidth versus quant tier, and whether to just get the 64GB box. No actual benchmark numbers are reported.
User runs Qwen3.6-35B-A3B-4bit on an M3 Max 128GB for production sub-agent delegations.
User also mentions GLM-5.1 for orchestration.
User is considering building a 5090 rig.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports 160 t/s prefill and 5-6 t/s generation on an M5 Max 128GB with Qwen 3.6 27B Q8 MLX at 290k context.
GPU utilization sits at only 36-50%.
User expected 8-14 t/s generation and asks how other setups compare.