- reported speed:
- 22.0 tokens/s generation
- quant:
- Q4
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports 8-22 t/s on a 32GB M4 MacBook Air with 21GB of allocations.
Setup is a custom inference engine called Cherenkov that combines predictive expert streaming with optional mixed-precision execution, keeping a bounded working set of experts in unified memory rather than loading the entire model.
A one-layer lookahead predicts which experts will be needed next and initiates SSD reads.
- reported speed:
- 1.8 tokens/s generation
- quant:
- JANG
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User benchmarks MoE streaming in oMLX on an M4 Pro 48GB across four models. GLM-5.3-Flash-JANG-MTP reaches 1.82 tok/s with 12.36s TTFT, 10.52 GiB after load and 14.68 GiB peak. Qwen3.8-JANG 4S reaches 3.63 tok/s with 8.56s TTFT, 7.04 GiB after load and 11.19 GiB peak. Qwen3.8-JANG 4M reaches 3.19 tok/s with 11.23s TTFT, 7.05 GiB after load and 11.11 GiB peak. DeepSeek-V4-Flash-0731-JANG reaches 2.71 tok/s with 6.52s TTFT, 8.25 GiB after load and 16.87 GiB peak.
The primary record uses GLM-5.3-Flash-JANG-MTP. The other models are Qwen3.8 in JANG 4S and 4M quants and DeepSeek V4 Flash in the 0731 variant with the JANG quant.
MoE streaming allows larger MoE models to run with a lower memory footprint at the cost of speed.
- reported speed:
- 25.0 tokens/s generation
- quant:
- UD-Q2_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports running a model in-browser with custom WebGPU kernels, at speed comparable to llama.cpp.
- reported speed:
- 18.0 tokens/s generation
- quant:
- 8-bit
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
mathcodingchat
User reports speculative decoding on Apple Silicon, with an 8-bit model going from 8.2 tok/s baseline to 18-26 tok/s.
The speedups are 3.27x on math, 2.5x on code and 2.22x on chat, with output byte-identical.
The 4-bit model gets about 1.7x at roughly 25 tok/s and needs about 18 GB, while the 8-bit model peaks at about 40 GB and needs a 48 GB Mac. Meta's DFlash numbers on Mac are 1.5x on an M4 Max and 1.8x on an M5 Max on 4-bit.
- reported speed:
- 8.2 tokens/s generation
- quant:
- 8-bit
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports mlx-dspark, a speculative decoding project, running an 8-bit model at 8.2 tok/s baseline and 18-26 tok/s with speculative decoding.
Speedups are 3.27x on math, 2.5x on code and 2.22x on chat. The 4-bit model reaches about 1.7x at about 25 tok/s and needs about 18 GB.
The 8-bit model peaks at about 40 GB and requires a 48 GB Mac. Output is byte-identical to normal decoding.
- reported speed:
- 20.0 tokens/s generation · 40.0 tokens/s prompt processing
- quant:
- ternary
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Maple Preview, a 20B A1B MoE model trained from scratch in ternary precision, running on a MacBook Air M4 with 16 GB RAM.
Setup is Mference, a fork of turbo-fieldfare, streaming experts from SSD and reducing memory usage to 500-1200 MB.
The model has limited world knowledge and multilingual capabilities but can use tools. The user is enthusiastic about the low memory footprint and potential for background agentic workloads.
- reported speed:
- 8.0 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User asks about an MTP or DFlash head for Qwen 3.8 27B, and mentions 35B-A3B as a daily driver.
User reports about 8 tok/s for Qwen 3.6 27B with MTP on 32 GB of unified memory.
- reported speed:
- 20.3 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
codingmath
User reports speculative decoding with mlx-dspark on an 8-bit target at a 2.45x mean speedup, 8.3 to 20.3 tok/s.
Setup is mlx-dspark with a 4-bit target at 1.74x and 25.3 tok/s.
The 8-bit target with drafter beats plain 4-bit at 14.6 tok/s.
- reported speed:
- 72.5 tokens/s generation · 1472.4 tokens/s prompt processing
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User benchmarks Qwen3.6-35B-A3B on a Mac with OMLX, comparing MTP enabled against disabled.
MTP shows minimal speedup for the 35B MoE model but roughly 2x for the 27B dense model.
Results include pp and tg t/s at various context lengths and batch sizes.
- reported speed:
- 11.0 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek-V4-Flash (284B) at about 11 t/s on a 64GB MacBook.
The model is 165GB on disk and does not fit in RAM, so it runs via SSD streaming with a 2-bit file.
Quality matches official perplexity (6.1250 vs 6.1262). Higher quality settings give 1-2 t/s. An 8GB cache gave 2.04 t/s versus 1.23 t/s with 32GB. Speculative decoding and prefetching did not help.
- reported speed:
- 36.0 tokens/s generation · 517.9 tokens/s prompt processing
- quant:
- AD-3.84bpw-M64 (gguf)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a 85 GB GGUF quant of Qwen 3.8 Flash Next running on a 64 GB MacBook with the ngram table offloaded to SSD, at 517.9 t/s prefill and 36 t/s decode.
The user notes the quant quality is far from perfect and better versions are planned.
- reported speed:
- 15.2 tokens/s generation
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
long-contextagentic
User benchmarks Qwen3.8 27B on an M4 Max 128GB across five runtime and quant stacks at contexts from 32K to 256K.
The stacks are oMLX AWQ 5-bit, oMLX oQ8e, mlx-dspark, and MTPLX 4-bit and 8-bit.
Best decode at 256K is oMLX AWQ 5-bit at 15.2 t/s, while MTPLX collapses at 256K. Prefix caching is crucial for effective speed.
User reports GLM 5.3 Flash support in the ds4 branch, running on an M4 Max with 128 GB.
No performance numbers are provided.
- quant:
- mixed-4_8bit (mlx)
codingmath
User reports Qwen 3.8 27B as the first model to break 94% on the cupel benchmark.
Setup is llama.cpp with the unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS quant, which outperformed other 4-bit quants.
Qwen 3.8 27B outperformed in coding but lost in general knowledge to Gemma 31B and Qwen 3.6.