Local model performance on your hardware

generation:

19.0 tokens/s

quant:

Q4_K_M (GGUF)

User compares Qwen3.8 27B variants on an M5 Max 36GB Mac Studio under Ollama 0.40.13, with single API requests on a 14k-token prompt at num_ctx 16384. Plain Q4_K_M in llama.cpp gives ~19 tok/s, nvfp4 in the MLX runner ~35-46 tok/s, and Q4_K_M with MTP ~36-38 tok/s (28 at 14k context). In real Cline agent sessions the MLX runner's held memory grows from 20 GB to 26.5 GB with peak 29.8 GiB in about 6 minutes, and swap reaches 8 GB of 9 GB. The llama.cpp runner with MTP holds ollama ps at 16 GB but the llama-server process reaches 19-20 GB RSS and swap grows from 1.7 to 4.4 GB. User links Ollama issues on prefix-cache accumulation and the llama-server host prompt cache (--cache-ram 8192 MiB) and plans to try LLAMA_ARG_CACHE_RAM=1024 or 0 via launchctl setenv, removing the mmproj, and LM Studio's MLX runtime or mlx_lm.server.

generation:

133.0 tokens/s

quant:

4-bit (MLX)

User benchmarks Qwen3.5 4B at 133 tok/s on an M5 Max 36GB, part of a 32-model comparison across four Macs. Setup is Rapid-MLX with 4-bit weights and no speculative decoding, 512-token prompts generating 128 tokens, median of 5 runs after one warmup. The same model reaches 63 tok/s on an M2 Pro 32GB, 79 on an M4 Pro 48GB and 159 on an M3 Ultra 256GB. On MoE models the M5 Max matches or beats the M3 Ultra, while the dense Qwen3.8 27B at 3.5bpw is faster on the Ultra (37 vs 28 tok/s). One 1024x1024 image with Qwen-Image 2.1 took 38 s on the M5 Max and 83 s on the M3 Ultra. The author works on Rapid-MLX and notes other runtimes will give different absolute numbers.

generation:

124.5 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.6-35B-A3B at 124.5 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is LM Studio's MLX backend with MLX 4-bit weights; configured context capacity, KV precision and Flash Attention are unknown. The figure is the article's rounded single-stream decode rate, measured as client-observed streaming decode-phase throughput rather than engine-native timing, and a separate four-stream run reached 265.6 t/s aggregate.

generation:

145.8 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.6 35B-A3B at 145.8 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is oMLX 0.7.0 with MLX 4-bit weights and an SSD cache directory enabled; KV precision, context capacity and speculative decoding are unknown. The 145.8 t/s is a rounded single-stream decode rate from a developer comparison, not engine-native timing; a separate four-stream run reached 281.9 t/s aggregate.

generation:

173.4 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

User reports Qwen3.6-35B-A3B at 173.4 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 default with MLX 4-bit weights and MTP speculative decoding on the text-only lane; configured context capacity, KV precision and Flash Attention are unknown. The figure is the article's rounded single-stream decode rate, client-observed streaming decode-phase throughput rather than engine-native timing, and the source warns differences under about 5% may be run variation.

generation:

81.1 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.5 9B at 81.1 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is LM Studio llmster 0.0.25 with the MLX runtime 1.11.0, MLX 4-bit weights, and unknown context capacity, KV precision and Flash Attention. The figure is the source's rounded single-stream decode rate, client-observed streaming decode-phase throughput rather than engine-native timing; a separate four-stream aggregate result is 230.0 t/s.

generation:

78.0 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.5 9B at 78 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is oMLX 0.7.0 with MLX 4-bit weights and an SSD cache directory enabled; configured context capacity, KV precision and speculative decoding are unknown. The figure is the source's rounded single-stream decode rate, client-observed streaming throughput rather than engine-native timing, and the source's homepage and prose give 77.9 t/s instead.

generation:

80.1 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3.5 9B at 80.1 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 with MLX 4-bit and --no-spec-decode, in image-capable mode benchmarked with text prompts. The 80.1 t/s is the article's rounded single-stream decode rate, measured as client-observed streaming decode throughput rather than engine-native timing; a separate four-stream run reached 77.0 t/s as an aggregate wall-throughput figure.

generation:

115.5 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

User reports Qwen3.5 9B at 115.5 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 with MLX 4-bit weights and MTP speculative decoding on the text-only lane, with image input disabled. The figure is the article's rounded single-stream decode rate, client-observed streaming decode-phase throughput rather than engine-native timing, and configured context capacity and KV precision are unknown.

generation:

26.0 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.8-27B at 26 t/s on an Apple M5 Max with 36 GB of unified memory. Setup is LM Studio with MLX 4-bit weights; KV cache and speculative decoding are not established. The figure is a single-stream decode rate, a median of three requests, not engine-native timing or prompt processing.

generation:

25.3 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.8-27B at 25.3 t/s on an Apple M5 Max with 36 GB unified memory. Setup is oMLX 0.7.0 with MLX 4-bit weights and an SSD cache directory enabled; speculative decoding is not established. The figure is a single-stream decode rate from a median of three requests, not engine-native timing or end-to-end throughput, and the source warns differences below roughly 5% can be run variation.

generation:

25.8 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 25.8 t/s on an M5 Max with 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 with MLX 4-bit weights and --no-spec-decode, running text prompts. The figure is a single-stream streaming decode rate, not engine-native timing or end-to-end throughput, and the harness requests a 512-token cap at temperature 0.

generation:

37.9 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

User reports Qwen3.8-27B at 37.9 t/s on an Apple M5 Max with 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 default with MLX 4-bit weights and MTP speculative decoding on the text-only lane, image input disabled. The figure is a single-stream decode rate, not end-to-end throughput or prompt processing, and comes from a median of three requests with a 512-token cap.

generation:

144.0 tokens/s

User reports Qwen3.8 27B at 144 t/s on an M5 Max MacBook Pro with 36 GB. Setup is the Inco Splash inference engine, an open-source engine built for Apple silicon, also available as a runtime inside LM Studio. Requirements are M3 or newer, macOS 26.4+, and 36 GB. The engine is claimed to reach up to 3x the decode speed of Ollama, 2x oMLX, and almost 4x when an agent fans out into sub-agents.