Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 27B at 48 tok/s decode and 198 tok/s prompt processing at 4K context on an M4 Max with 36GB unified memory.
Setup is Ollama 0.34.4 with NVFP4/MLX weights, q8_0 KV cache, flash attention enabled, and a 128K configured context. The GGUF Q4_K_M version of the same model reached only about 10.7 tok/s decode.
Decode falls to 44 tok/s at 8K, 25 tok/s at 16K and 32K, and 14 tok/s at 115K, where time to first token reaches 21 minutes. The user notes that maximum context is not the same as comfortable interactive context and that memory pressure and swap grew during the stress runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.6 27B at 18.3 tok/s decode on an Apple M4 Pro with 48 GB unified memory.
Setup is MLX with 4-bit weights, 4096 context, and native MTP speculative decoding at depth 3.
Without MTP the model decodes at about 7 tok/s; served over WiFi it reaches 14.3 tok/s, and llama.cpp with MTP gives 10.5 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-use
User reports Qwen3.5 4B at ~85 tok/s decode on an M4 Max 48GB, running a Mac meeting notetaker stack.
Setup is llama.cpp with Q4_K_M weights; the full on-device stack totals 6.8 GB across five models.
A 26-minute meeting produced finished notes in 33 s warm. Speaker error dropped from 43.4% to 14.6% DER on AMI, and autocomplete p95 improved from 1.83 s to 0.49 s. MiniCPM5 2B was tried for notes but repeated itself and misassigned action items, so Qwen3.5 4B stayed the default.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 53.3 tok/s on a Mac Studio M4 Max 128GB.
Setup is oMLX 0.6.3rc2 with oQ4e 4-bit affine weights, 262,144 context, ANE prefill and native MTP speculative decoding with k=3 draft tokens, single stream.
Code generation measured 72.1 tok/s and prefill at 4K 273.7 tok/s. oMLX 0.6.1 with MTP gave 48.0 tok/s prose and 65.5 tok/s code; oMLX 0.6.3rc2 with ANE off gave 47.9 tok/s for both; vllm-metal 0.3.0 at 8-bit without MTP gave 13.2 tok/s code.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 24.8 tok/s decode on an M4 Pro 48GB, with median TTFT 943 ms over three samples of 4096 prompt tokens and 128 generated tokens.
Setup is oMLX 0.6.4 with the oQ4e 4-bit MLX model and Lightning MTP draft depth 3 enabled via the mtp3 profile, at a 200k context window.
Enabling MTP through a profile rather than settings.json raised decode from 10 tok/s to 24.8 tok/s. A separate sweep documents the MTP-on prefill curve at 1.8k-30k prompts and a 5-turn agentic session reaching 157k context with 85% token reuse.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticvisionlong-context
User reports Qwen3.8-Flash-Next at 52.6 tok/s decode and 762 tok/s prefill on an M4 Max 64GB.
Setup is mlx-serve with an iQ-MLX 3.3bpw pack, 64k context, 52 GB resident; the 32 GB n-gram table is mmapped and not resident.
The same run measured the mixed-4-8bit pack at 55.5 tok/s decode, 754 tok/s prefill and 278 ms TTFT; MTP is in the pack but default-off and not re-measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Phi-3.5-mini-instruct-4bit at 72.2 t/s on a Mac Mini M4 Pro 64GB.
Setup is mlx-lm 0.30.7 with 4-bit weights and kv4 KV-cache quantization at 4096 generated tokens.
KV-cache quantization with kv4 uses 0.51 GB and is 1.1% faster than unquantized (71.4 t/s), while kv8 uses 0.91 GB and is 4.9% slower (67.9 t/s). User also tested Qwen2.5-14B-Instruct-4bit and found GQA halves KV-cache per token versus Phi-3.5-mini's MHA.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-30B-A3B 4-bit at 127.7 tok/s on an M4 Max 128GB.
Setup is vllm-mlx, a native MLX inference server, with greedy decoding and single-stream decode.
Two other models were also measured on the same machine: Qwen3-0.6B 8-bit at 417.9 tok/s and Llama-3.2-3B-Instruct 4-bit at 205.6 tok/s. Continuous-batching results, KV-cache quantisation and MoE top-k sweeps are documented separately.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks three Qwen3.8-Flash-Next REAP builds for Apple Silicon on an M4 Max 40-core GPU with 128 GB unified memory at 4K context.
Setup is oMLX with mixed-precision packages, 48 transformer layers, top-10 routing, a retained 512-expert native MTP predictor, and 262k configured context. The REAP-384 oQ5e build scored 97/100 HumanEval no-thinking and 61/100 LiveCodeBench no-thinking at 679.1 PP tok/s, 53.9 TG tok/s, and 72.8 GB peak memory.
The REAP-384 oQ4e build reached 689.5 PP tok/s and 59.3 TG tok/s at 61.5 GB peak, while REAP-288 oQ4e reached 692.3 PP tok/s and 48.7 TG tok/s at 46.0 GB peak. Accuracy figures are 100-question stratified samples and throughput is workload-specific.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8-22 t/s on a 32GB M4 MacBook Air with 21GB of allocations.
Setup is a custom inference engine called Cherenkov that combines predictive expert streaming with optional mixed-precision execution, keeping a bounded working set of experts in unified memory rather than loading the entire model.
A one-layer lookahead predicts which experts will be needed next and initiates SSD reads.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks MoE streaming in oMLX on an M4 Pro 48GB across four models. GLM-5.3-Flash-JANG-MTP reaches 1.82 tok/s with 12.36s TTFT, 10.52 GiB after load and 14.68 GiB peak. Qwen3.8-JANG 4S reaches 3.63 tok/s with 8.56s TTFT, 7.04 GiB after load and 11.19 GiB peak. Qwen3.8-JANG 4M reaches 3.19 tok/s with 11.23s TTFT, 7.05 GiB after load and 11.11 GiB peak. DeepSeek-V4-Flash-0731-JANG reaches 2.71 tok/s with 6.52s TTFT, 8.25 GiB after load and 16.87 GiB peak.
The primary record uses GLM-5.3-Flash-JANG-MTP. The other models are Qwen3.8 in JANG 4S and 4M quants and DeepSeek V4 Flash in the 0731 variant with the JANG quant.
MoE streaming allows larger MoE models to run with a lower memory footprint at the cost of speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running a model in-browser with custom WebGPU kernels, at speed comparable to llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports mlx-dspark, a speculative decoding project, running an 8-bit model at 8.2 tok/s baseline and 18-26 tok/s with speculative decoding.
Speedups are 3.27x on math, 2.5x on code and 2.22x on chat. The 4-bit model reaches about 1.7x at about 25 tok/s and needs about 18 GB.
The 8-bit model peaks at about 40 GB and requires a 48 GB Mac. Output is byte-identical to normal decoding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Maple Preview, a 20B A1B MoE trained from scratch in ternary precision, generating at around 20 tokens/s on a MacBook Air M4.
Setup is the user's Maple integration in Mference, a fork of turbo-fieldfare that keeps the weights on SSD and streams MoE experts from disk, using between 500 and 1,200 MB of memory depending on context length.
Prompt processing runs at around 40 tokens/s, although the title gives 40 tps generation. On Tau-2 the user measured 0.48 Airline, 0.175 Retail and 0.427 Telecom.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks about an MTP or DFlash head for Qwen 3.8 27B, and mentions 35B-A3B as a daily driver.
User reports about 8 tok/s for Qwen 3.6 27B with MTP on 32 GB of unified memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports speculative decoding with mlx-dspark on an 8-bit target at a 2.45x mean speedup, 8.3 to 20.3 tok/s.
Setup is mlx-dspark with a 4-bit target at 1.74x and 25.3 tok/s.
The 8-bit target with drafter beats plain 4-bit at 14.6 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B on a Mac with OMLX, comparing MTP enabled against disabled.
MTP shows minimal speedup for the 35B MoE model but roughly 2x for the 27B dense model.
Results include pp and tg t/s at various context lengths and batch sizes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash (284B) at about 11 t/s on a 64GB MacBook.
The model is 165GB on disk and does not fit in RAM, so it runs via SSD streaming with a 2-bit file.
Quality matches official perplexity (6.1250 vs 6.1262). Higher quality settings give 1-2 t/s. An 8GB cache gave 2.04 t/s versus 1.23 t/s with 32GB. Speculative decoding and prefetching did not help.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a 85 GB GGUF quant of Qwen 3.8 Flash Next running on a 64 GB MacBook with the ngram table offloaded to SSD, at 517.9 t/s prefill and 36 t/s decode.
The user notes the quant quality is far from perfect and better versions are planned.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextagentic
User benchmarks Qwen3.8 27B on an M4 Max 128GB across five runtime and quant stacks at contexts from 32K to 256K.
The stacks are oMLX AWQ 5-bit, oMLX oQ8e, mlx-dspark, and MTPLX 4-bit and 8-bit.
Best decode at 256K is oMLX AWQ 5-bit at 15.2 t/s, while MTPLX collapses at 256K. Prefix caching is crucial for effective speed.