Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports decode speed improvements on Qwen3-30B-A3B with IQ3_XXS quantization on an M5 MacBook Pro 24GB. The decode speed increased from 65.6 to 73.9 tok/s using a llama.cpp PR for Metal optimization.
Setup uses llama.cpp with IQ3_XXS quantization. The user also tested on an M1 Pro 32GB.
The user is looking for testers for Qwen3.8-Flash-Next and provides benchmark commands.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks oMLX 0.7.0 against 0.7.0rc1 on an M5 Ultra 256GB, running GLM-5.3-Flash oQ4 at 4K–200K context. Prefill improves from 792 to 2,270 tok/s (median) and decode from 55.4 to 76.7 tok/s. A 1M-token prompt completes with prefill 1,485 tok/s, decode 41.5 tok/s, and time to first token 11.8 min. Setup uses Lightning MTP, TurboQuant KV 4-bit, and one model loaded at a time.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Swift-1.5 at 113.7 tok/s decode and 3191 tok/s prefill on an M5 Ultra 96GB Mac Studio.
Setup is mlx-serve 26.10.1 with a 4.7bpw MLX quantization, 107GB download, text-only, tested to 179,200 tokens of context.
Decode falls to 81.1 tok/s after a 95k prompt, where prefill is 2928 tok/s. User compares against a llama.cpp IQ3_XXS build of the same model, which reaches 62.7 tok/s decode after 4k and 1427 tok/s prefill at 25k. Top-1 agreement with Swift BF16 is 91.0% across 680 held-out positions, against 84.1% for the llama.cpp build.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports GLM-5.3 Flash at 68.8 tok/s on an M5 Ultra 256GB.
Setup is oMLX 0.7.0 with oQ4e quantization and MTP speculative decoding, with more optimization still to come.
Prefill is 1,878 toks. User shared the result because other benchmarks looked lower than expected.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6-35B-A3B at 105 tok/s decode at 128 context and 97 tok/s at 8k on an M5 Max 128 GB.
Setup is llama.cpp with Q8_0 GGUF weights, MTP speculative decoding on, 65536 context, 45 GB memory.
Without MTP the same model gives 93 tok/s at 128 and 87 tok/s at 8k; MLX gives 85 and 79 tok/s. MTP adds 75% on the dense 27B (17 to 32 tok/s) but only 12% on the MoE 35B-A3B. llama.cpp beats MLX by 10-24% on this machine. DeepSeek-V4-Flash under ds4 at q2-q4 (91 GB) reaches 33 tok/s at 128 and 28 tok/s at 8k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen 3.8 Flash-Next (125B-A6B MoE) at 154.5 tok/s decode and 4,183 tok/s prefill on a single Mac Studio M5 Ultra 256 GB.
Setup is mlx-serve 26.9.7-dev with the MLX-Serve mixed 4/8-bit pack (100 GiB), KV cache bf16, MTP speculative decoding, PLD and batched decode, at 262,144 context.
Decode holds ~150 tok/s through 66k and is 114.4 tok/s at 256k; prefill stays flat at ~4.1-4.35k tok/s. A 4-stream burst gives 166.8 tok/s aggregate versus 119.1 alone. The user notes long-context recall was not verified because answers never cited the planted value.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks dflash-mlx at 121.2 t/s decode on a 32,768-token prompt, the fastest of four MLX frameworks tested on an Apple M5 Max with 64 GB unified memory.
Setup is dflash-mlx v0.1.7 with the 4-bit MLX Qwen3.6-35B-A3B model and a companion draft model for speculative decoding, single sequence, prefix caching disabled, cooldown=60 between runs.
Decode medians across the ladder: 148.7 t/s at 64 tokens, 138.9 at 512, 154.5 at 2,048, 127.7 at 4,096, 124.6 at 8,192, 119.0 at 16,384, and 121.2 at 32,768. TTFT at 32K is 8.1 s. The other three frameworks were tested at cooldown=2 and would score higher at the longer cooldown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User benchmarks DeepSeek-Coder-V2-Lite-Instruct at 125.8 tok/s generation and 363.9 tok/s prompt processing on an Apple M5 Pro with 24GB unified memory.
Setup is MLX with a 4bit quant (8.2GB) on a single machine, running a fixed coding task capped at 1500 output tokens.
The model produced correct working code; the user notes that 24GB has a hard ceiling around 20GB models and that background downloads measurably affect tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 47.0 t/s generation and 854 t/s prompt processing on an Apple M5 Max with 128 GB unified memory.
Setup is mlx-vlm with the 4-bit MLX quantisation and MTP speculative decoding enabled, 15.5 GB resident. The 8-bit quant reaches 28.7 t/s generation and 883 t/s prompt processing at 28.1 GB resident.
MTP speculative decoding is reported as worth 2.2x (15.5 t/s to 34.1 t/s at 8-bit). Decode falls to 36.1 t/s at 8K context, and time to first token climbs to about 14 s at that depth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User asks whether oMLX benchmarks can be spoofed, citing a third-party oMLX benchmark entry for deepseek-v4.1-flash-q3-g128-lsq-mlx on an M5 Ultra with 256GB RAM, posted two days earlier, reporting about 73.4 tok/s generation and 1,010 tok/s prompt processing.
The run is not the user's own; the figures come from a linked oMLX benchmark page and screenshot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodinglong-context
User benchmarks Qwen3.8-Flash-Next 8-bit on a Mac Studio M5 Ultra 256GB, reporting 17 t/s decode and 1,069 t/s prefill at 250k context on llama.cpp (Metal).
Setup is llama.cpp with Q8_0 weights and MTP off; oMLX 0.7.0rc1 reaches 51 t/s decode and 1,365 t/s prefill at the same depth, and 88 t/s decode at 10k with MTP on.
A cold 250k-token prompt took 234 s before the first token, while the cached next step took 0.64 s. DeepSeek-V4-Flash at 400k and 500k context took 934 s and 1,311 s cold. The user notes one overnight agent run per setup and that the DeepSeek engine comparison mixes two model variants.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-Flash-Next 8-bit at 29 t/s decode on a 1M-token prompt on an M5 Ultra 256GB Mac Studio.
Setup is oMLX 0.7.0 with oQ8e weights and YaRN x4 position scaling; the 1M-token first read took 5.3 minutes and the next turn 6.7 s, finding all three hidden codes.
At 250k context the model prefills at 4,235 t/s and decodes at 63 t/s; the user notes YaRN stretches a 262k-trained model and that a needle test does not prove reasoning at 1M.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathagentic
User reports Qwen3.8-Flash-Next at 40.8 tok/s on an M5 Pro 64GB Mac using the Slipstream engine.
Setup is Slipstream with Q4_0 GGUF weights and SSD expert streaming, running the 95.5 GiB model on a single 64GB machine.
The figure is the average across six reasoning and coding tasks, compared to 23.1 tok/s in a llama.cpp fork (1.76x speedup). Decode stays between 33 and 44 tok/s out to 130,000 tokens across 3,086 live requests. An optional Swift KV-sparse variant scores 70.3% overall on 145 paired evaluation problems versus 67.6% for the original.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-30B-A3B 4-bit at 57 tok/s cold generation on an M5 MacBook Air with 32GB.
Setup is MLX with mlx-lm, 4-bit weights, 256 generated tokens, no speculative decoding.
Sustained generation drops to 44 tok/s on battery and 39 tok/s on mains; prompt processing of a 4,096-token prompt is 1,063 tok/s cold, 788 sustained on battery and 725 on mains. The decline finishes at about 150 seconds and the rate then holds flat.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8-Flash-Next against Laguna-S-2.1 on a Mac Studio M5 Ultra 256GB, both capped at 262K context with thinking on and unique content per run.
Qwen runs under oMLX with an oQ8e quant and MTP speculation, holding roughly 4,200 tok/s prefill and 59-74 tok/s decode across sizes. Laguna runs under LM Studio with an 8-bit quant, prefill degrading superlinearly from 10.3s at 8K to 455.4s at 200K, and decode falling from 68 to 34 tok/s with no speculation.
Quality was a draw at 4/4 each on four script-verified problems. At 200K, prefill is 94% of total time on both. The user notes an earlier run was invalidated by shared prefixes letting the KV cache carry over.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 40-60 t/s on an M5 Pro Mac mini with 64GB RAM using the Tensorfold engine.
Setup is Tensorfold with the Vontra/Qwen3.8-27B-MLX-4bit model and z-lab/Qwen3.8-27B-DFlash2 draft model for speculative decoding.
The user compares this to an RTX 3090 Ti running Qwen3.8-27B-GSQ-RCO-GGUF with MTP IQ3_S at 50-70 tok/s, noting the 3090 Ti has roughly 4x faster prefill. They are considering replacing the 3090 Ti with the Mac mini as their main inference server.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 1900 t/s prefill and 90 t/s generation on an M5 Max 128GB.
Setup is mlx-serve with Sushi-4 quant, an MLX-EXL3 hybrid, with MTP and Vision supported.
User also cites M1 Max 64GB at ~350 t/s prefill and 30 t/s gen, and M5 Pro at ~900 t/s prefill and 60 t/s gen.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 35B A3B running at roughly 17 t/s on a MacBook Pro M5 16GB.
Setup uses slotstream; no engine, quant or context length stated.
The figure is given as approximate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 50-70 t/s on an M5 Max 128GB after upgrading to macOS 27, down from 60-90 t/s on macOS 26.
Setup is oMLX with oQ5e quantization.
The user reports roughly 20% slower token generation after the OS upgrade and asks whether others see the same.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at 108 t/s generation and 2,887 t/s prompt processing on an M5 Ultra 256GB Mac Studio.
Setup is oMLX 0.7.0.dev2 with MLX backend, oQ4e 4-bit quant, multi-token prediction depth 3, 16K prompt.
M3 Ultra 512GB comparison: 70 t/s generation, 1,143 t/s prompt. At 256K context, M5 prompt processing was 2,544 t/s. Qwen3.8-27B on M5 Ultra: 48 t/s generation, 1,701 t/s prompt; RTX 5090 PC in LM Studio: 59 t/s generation, 3,031 t/s prompt.