Local model performance on your hardware

generation:

63.0 tokens/s

quant:

4-bit

User benchmarks Kapitsa 0.2 with its KFlash speculative decoding at 63 tok/s on Qwen3.8 27B on a MacBook Pro M4 Pro 48GB. Setup is Kapitsa with 4-bit weights, decoding a 512-token coding answer at temperature 0.6, median of 5 runs. The post also cites Splash 1.4.0 (DFlash2) at about 50 tok/s and Kapitsa at about 65 tok/s in its title, and mentions an M1 Pro 16GB MacBook Pro and Qwen3.8 4B/35B runs that are not shown in the excerpt.

generation:

154.0 tokens/s

quant:

4bit (MLX)

User reports Qwen3.6 35B-A3B at 154 t/s on a 2024 MacBook Pro M4 Pro with 48GB, using the Kapitsa engine. Setup is Kapitsa, an open-source Metal engine for Apple Silicon, with 4-bit MLX weights and DFlash2 speculative decoding; part of the 27B's prefill runs on the Neural Engine. The same table gives Qwen3.8 27B at 63 t/s and Qwen3.5 4B at 179 t/s on the M4 Pro, and on an M5 Max 128GB Qwen3.6 35B reaches 281 t/s. The author says not to trust benchmarks and notes Kapitsa is not fastest on the M5 Max for Qwen3.8 27B (102 vs mlx-serve's 130) or Flash-Next.

generation:

63.0 tokens/s

quant:

4bit (MLX)

User reports Qwen3.8 27B at 63 tok/s on an M4 Pro 48GB MacBook Pro with the Kapitsa engine, ahead of mlx-serve at 54 and Ollama at 32. Setup is Kapitsa with 4-bit MLX weights and speculative decoding (DFlash2), with part of each layer's prefill on the Neural Engine. The post is an engine release with comparison tables across M4 Pro 48GB, M5 Max 128GB and M1 Pro 16GB; on the M5 Max, Qwen3.8 27B reaches 115 tok/s and Qwen3.6 35B-A3B 272 tok/s, while Flash-Next 180B-A7B at 105 tok/s trails mlx-serve and oMLX by 2-3%.

generation:

198.4 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

User reports 198.4 t/s generation. Source evidence (verbatim): Source: https://rapidmlx.com/compare/mlx-lm # Mac mini M4 Pro48GB Qwen3.5-9B speculative whole-file edit benchmark First-party engine-author measurements October4,2026. Mac mini AppleM4 Pro 48GB unified RAM, macOS26.5.1, ACpower, idle before starts; core counts unknown. Qwen3.5-9B4-bit, Rapid-MLX0.15.6 with MTP plus prompt-lookup versus mlx-lm0.32.0 default nonspeculative server, both MLX0.32.3. Whole-file rename task: 198.4 tok/s decode. Comparison baseline: 46.4 tok/s; median of3fresh-server runs, two discarded warmups, temperature0, thinkingoff. Exact prompt/output counts, contextcapacity, residentRAM and MTPcompanion artifact unknown. Decode=(completiontokens-1)/(last-firststreamedchunk), not end-to-end or prefill. Copy-heavy task benefits from prompt lookup; do not generalize peak. Testedquant repo verified: https://huggingface.co/mlx-community/Qwen3.5-9B-4bit . Sourceversions.json identifies shortened revision8b2b98c; fullSHA unknown and revisiontree read returned generic toolerror. Sources: https://rapidmlx.com/compare/mlx-lm ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/versions.json ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/raw/manifest.json . Source author builds tested engine; rawmanifest links individual runs. Selected run is Rapid-MLX with MTP plus prompt-lookup, generation 198.4 tok/s. Baseline mlx-lm measurement is a separate run. Prefill speed unknown.

generation:

77.2 tokens/s

quant:

MXFP4-Q8 (MLX)

User reports gpt-oss 21B-A3.6B at 77.2 t/s on an M4 Pro 48GB. Setup is Rapid-MLX 0.15.4 with MXFP4-Q8, single-stream generation, median of three runs at temperature 0 over a 512-token decode workload. Four concurrent streams deliver 126.7 t/s aggregate, and the reported 15.7 GB Metal peak covers the full sweep, including up to approximately 16K prompt and four streams with prefix caching.

generation:

78.5 tokens/s

quant:

4-bit (MLX)

User reports Gemma 4 26B-A4B at 78.5 t/s on an M4 Pro with 48 GB unified memory. Setup is Rapid-MLX 0.15.4 with MLX 4-bit on one Mac mini, vision disabled, decoding a 512-token workload. The figure is single-stream, a median of three runs at temperature 0; a separate four-stream result of 121.6 t/s is aggregate and not single-request speed.

generation:

90.0 tokens/s

quant:

oQ4 (MLX)

User reports Qwen3.6 35B-A3B at 90 tok/s on an M4 Pro Mac mini with 48GB unified memory. Setup is oMLX 0.7.0 with a custom oQ4 quantised model using MTP headers. User notes performance had recently fallen as low as 10-11 tok/s before this improvement.

generation:

25.5 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.8-27B Uncensored at 25.48 tok/s with DFlash2 speculative decoding on an M4 Pro 48GB, a 2.06x speedup over the 12.35 tok/s baseline. Setup is dflash 0.1.0 on MLX with a 4-bit MLX target model and a 4-bit draft, block size 5, greedy decoding on HumanEval with 20 samples and 256 max new tokens. Mean acceptance length was 4.87 tokens with 94.9% of blocks accepted in full. An initial 3-sample smoke test gave 13.76 tok/s baseline and 29.11 tok/s with DFlash. Weights and caches were on an external HDD; the 825.84s wall time includes model loading and is not generation throughput.

generation:

104.5 tokens/s

quant:

MLX 4-bit (MLX)

User reports Qwen3.6-35B-A3B at 104.5 tokens/s single-stream decode on an Apple Mac mini M4 Pro with 48 GB unified memory. Setup is Rapid-MLX 0.15.4 with MLX 4-bit weights (17.5 GB on disk). The figure is the median of three runs at temperature 0. Separately, the source reports 106.5 tokens/s aggregate with four concurrent streams and a 2.4 s TTFT for a 2K-token prompt. The 23.2 GB Metal peak covers the whole sweep, not single-stream use. These are developer-produced measurements; the linked raw-data directory returned a 404 when checked.

generation:

60.1 tokens/s

quant:

MLX 4-bit (MLX)

mtp (multi-token prediction):

on

User reports Qwen3.5-9B at 60.1 tokens/s median decode on an Apple Mac mini M4 Pro with 48 GB unified memory. Setup is Rapid-MLX 0.15.6 with mlx-community/Qwen3.5-9B-4bit, greedy decoding, thinking off, MTP plus prompt lookup speculation, and a short-story task. Baseline mlx-lm 0.32.0 server with speculation off reaches 47.1 tokens/s. Three-run medians with fresh server and isolated cache per run. The whole-file rename task reaches 198.4 tokens/s versus 46.4 for the baseline, but that peak is task-specific and not a general generation rate.

generation:

150.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.5-4B think at ~150 t/s on an M4 Pro 48GB. Setup is llama.cpp with Q4_K_M at 32k context, using 2.5 GB of RAM. The model scores 6/7 on the CC-Agent harness and 14/14 on the V3.1 text suite, matching the 35B-A3B model at 1/8 the memory.

generation:

18.3 tokens/s

quant:

4-bit (MLX)

User reports Qwen 3.6 27B at 18.3 tok/s decode on an Apple M4 Pro with 48 GB unified memory. Setup is MLX with 4-bit weights, 4096 context, and native MTP speculative decoding at depth 3. Without MTP the model decodes at about 7 tok/s; served over WiFi it reaches 14.3 tok/s, and llama.cpp with MTP gives 10.5 tok/s.

generation:

24.8 tokens/s

quant:

oQ4e (MLX)

User reports Qwen3.8-27B at 24.8 tok/s decode on an M4 Pro 48GB, with median TTFT 943 ms over three samples of 4096 prompt tokens and 128 generated tokens. Setup is oMLX 0.6.4 with the oQ4e 4-bit MLX model and Lightning MTP draft depth 3 enabled via the mtp3 profile, at a 200k context window. Enabling MTP through a profile rather than settings.json raised decode from 10 tok/s to 24.8 tok/s. A separate sweep documents the MTP-on prefill curve at 1.8k-30k prompts and a 5-turn agentic session reaching 157k context with 85% token reuse.

generation:

1.8 tokens/s

quant:

JANG

User benchmarks MoE streaming in oMLX on an M4 Pro 48GB across four models. GLM-5.3-Flash-JANG-MTP reaches 1.82 tok/s with 12.36s TTFT, 10.52 GiB after load and 14.68 GiB peak. Qwen3.8-JANG 4S reaches 3.63 tok/s with 8.56s TTFT, 7.04 GiB after load and 11.19 GiB peak. Qwen3.8-JANG 4M reaches 3.19 tok/s with 11.23s TTFT, 7.05 GiB after load and 11.11 GiB peak. DeepSeek-V4-Flash-0731-JANG reaches 2.71 tok/s with 6.52s TTFT, 8.25 GiB after load and 16.87 GiB peak. The primary record uses GLM-5.3-Flash-JANG-MTP. The other models are Qwen3.8 in JANG 4S and 4M quants and DeepSeek V4 Flash in the 0731 variant with the JANG quant. MoE streaming allows larger MoE models to run with a lower memory footprint at the cost of speed.

generation:

8.2 tokens/s

quant:

8-bit

User reports mlx-dspark, a speculative decoding project, running an 8-bit model at 8.2 tok/s baseline and 18-26 tok/s with speculative decoding. Speedups are 3.27x on math, 2.5x on code and 2.22x on chat. The 4-bit model reaches about 1.7x at about 25 tok/s and needs about 18 GB. The 8-bit model peaks at about 40 GB and requires a 48 GB Mac. Output is byte-identical to normal decoding.

generation:

20.3 tokens/s

User reports speculative decoding with mlx-dspark on an 8-bit target at a 2.45x mean speedup, 8.3 to 20.3 tok/s. Setup is mlx-dspark with a 4-bit target at 1.74x and 25.3 tok/s. The 8-bit target with drafter beats plain 4-bit at 14.6 tok/s.