Local model performance on your hardware

generation:

36.0 tokens/s

quant:

MLX 4-bit (MLX)

User compares six local runtimes on an Apple M5 Pro with 64 GB of unified memory, all running Qwen3.8-27B with a short lighthouse-keeper prompt of about 44 input tokens and 400 output tokens. TensorFold app beta.8 with engine 1.1.0 and an MLX 4-bit model plus DFlash2 drafter was fastest at a mean 36.0 tok/s (36.2 and 35.7 across two runs). Other means were LM Studio 0.4.26+4 at 33.5 tok/s, MLX-Serve 26.10.2 at 30.1 tok/s, Ollama 0.40.6 at 27.0 tok/s, Magnitude 0.2.9 at 21.0 tok/s and oMLX 0.7.0 at 16.0 tok/s. User notes the comparison is not apples-to-apples since model packaging, quantization, speculative decoding and runtime defaults differ, and that TensorFold's hardware qualification does not list the M5 Pro.

generation:

98.2 tokens/s

quant:

MLX 4-bit affine, group 64 (MLX)

User reports Qwen3 30B-A3B at 98.2 t/s on an M5 Pro with 64GB unified memory. Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1, 944-token prompt and 512 output tokens. The figure is the arithmetic mean of six fresh-process runs, ranging 97.97 to 98.37 t/s; an altered 6-expert run reached 107.79 t/s but is not a standard-model result.

generation:

18.5 tokens/s

prompt processing (prefill):

381.6 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8 27B at 18.5 t/s generation and 381.6 t/s prompt processing on an M5 Pro with 64GB unified RAM. Setup is llama.cpp with the Unsloth UD-IQ4_XS quant, run via llama-bench at -p 512 -n 128. The source gives 18.45 +/- 0.03 tok/s and 381.62 +/- 0.18 tok/s; the generation figure is not a post-512-prompt end-to-end run, and context limit, KV precision and offload settings are unstated.

generation:

15-17.2 tokens/s

prompt processing (prefill):

500-600 tokens/s

quant:

FP8 (FP8)

User reports Qwen3.8-Flash-Next (FP8) at 15.0–17.2 tok/s decode and 500–600 tok/s prefill at 16K context on an M5 Pro 64GB Mac. Setup is Whallm 1.1.11 with MTP on by default, streaming experts from SSD; peak MLX memory stays under 19 GiB at 16K context. A second row for Swift1.5-Qwen3.8 (MXFP4) gives 14.6–17.5 tok/s decode and 571 tok/s prefill at 18.8 GiB peak. Previous v1.1.7 numbers were ~9 tok/s decode and ~146 tok/s prefill.

generation:

0.5-0.6 tokens/s

quant:

VQ3R

User reports Kimi K3 2.78T running at 0.45-0.62 tok/s on a 64 GB MacBook Pro with M5 Pro, streaming experts from internal NVMe. Setup is the WARP engine with VQ3R 3-bit expert quantization, a 46.39 GB memory budget including a 17.56 GB expert cache, and a 982 GB converted container. The 64-token run is a range; the 200-token column is absent because one run takes ten minutes and reads 4.6 TB. The engine also runs DeepSeek-V4.1-Flash 552B at 3.77/3.71 tok/s and GLM-5.3-Flash 313B at 3.32/3.86 tok/s on the same machine.

generation:

35.3 tokens/s

prompt processing (prefill):

528.9 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks Qwen3.8 27B at 35.28 t/s decode and 528.91 t/s prefill on an M5 Pro 64GB Mac Mini. Setup is Splash with Q8_0 GGUF weights and 4k input prompts. User also compares llama.cpp, MLX, and Splash across several quants, and reports an M1 Max 64GB Splash run at 135.96 t/s decode and 915.01 t/s prefill. User settled on Splash with Q8_0 and is happy with the results.

generation:

40.8 tokens/s

quant:

Q4_0 (GGUF)

User reports Qwen3.8-Flash-Next at 40.8 tok/s on an M5 Pro 64GB Mac using the Slipstream engine. Setup is Slipstream with Q4_0 GGUF weights and SSD expert streaming, running the 95.5 GiB model on a single 64GB machine. The figure is the average across six reasoning and coding tasks, compared to 23.1 tok/s in a llama.cpp fork (1.76x speedup). Decode stays between 33 and 44 tok/s out to 130,000 tokens across 3,086 live requests. An optional Swift KV-sparse variant scores 70.3% overall on 145 paired evaluation problems versus 67.6% for the original.

generation:

40-60 tokens/s

quant:

4bit (MLX)

User reports Qwen3.8-27B at 40-60 t/s on an M5 Pro Mac mini with 64GB RAM using the Tensorfold engine. Setup is Tensorfold with the Vontra/Qwen3.8-27B-MLX-4bit model and z-lab/Qwen3.8-27B-DFlash2 draft model for speculative decoding. The user compares this to an RTX 3090 Ti running Qwen3.8-27B-GSQ-RCO-GGUF with MTP IQ3_S at 50-70 tok/s, noting the 3090 Ti has roughly 4x faster prefill. They are considering replacing the 3090 Ti with the Mac mini as their main inference server.

generation:

23.0 tokens/s

prompt processing (prefill):

368.3 tokens/s

quant:

4bit (MLX)

User reports Qwen3.8 27B at 22.99 tokens/s three-scenario decode average on an Apple M5 Pro with 64 GB of unified memory. Setup is oMLX 0.6.0 with 4-bit affine MLX weights and an external Qwen3.8-27B-MTP-4bit drafter (250.9 MB, block size 3), measured at 16384 tokens of context. The average combines 26.28 short, 22.81 long, and 19.88 at 11k context. First uncached 11k prefill was 368.32 tokens/s with 30.18 s TTFT. MTP accepted 1,836 of 3,416 drafted tokens (53.7%).

generation:

50.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8 27B at 50 t/s on an M5 Pro 64GB. Setup is Inco Splash with Unsloth UD-Q4_K_XL GGUF weights, run in an agentic setup. The user credits Splash's optimized kernels, speculative decoding, prefix cache and mixed-weight support, and calls it a breakthrough in local inference on Apple Silicon.

generation:

36.9 tokens/s

quant:

Q8 (Splash)

User reports Qwen3.8 27B at 36.9 t/s average on an M5 Pro 64GB using the Splash engine with native 8-bit weights. Setup is Splash with speculative decoding, 27 GB native 8-bit model, evaluated at temperature 0.0 across five task domains; speeds range from 21.9 to 54.8 t/s. Context scaling tested to 190,016 tokens with decode speed staying between 21 and 33 t/s; cold prefill at 180k+ tokens takes 4-5 minutes.

generation:

27.6 tokens/s

prompt processing (prefill):

367.0 tokens/s

quant:

Q4_0 (GGUF)

User reports Qwen3.8 Flash-Next at ~27.6 t/s generation and ~367 t/s prompt processing on an M5 Pro 64GB, with the 95.5 GiB model streamed from SSD. Setup is a llama.cpp fork with expert streaming, a draft head, depth 3 with a 0.3 confidence floor, sparse attention, and Metal MoE fusion, using a bartowski Q4_0 checkpoint with Q8_0 output layer and unsloth UD-IQ4_XS spliced in. Generation drops to 18-18.6 t/s with the draft head off and ~20.6 t/s at 29k context in real chat use. Paired 40-chunk perplexity improved from 5.2777 to 4.3148.

generation:

20.0 tokens/s

User reports Qwen3.8 27B at around 20 t/s sustained on an M5 Pro 64GB, with about 30 t/s in the first 2-4k tokens. User finds it unbearably slow, possibly because the model thinks so much. User also tried antirez's DS4 with DeepSeek V4 Flash and gets around 8 t/s sustained, which they call slow and basically unusable.

generation:

15.0 tokens/s

quant:

Q8_0 (GGUF)

User reports 10-17 t/s in SSD streaming mode. Setup keeps routed experts partly in RAM cache and pulls them from the GGUF on cache misses. Experts and output head are Q8_0, while router, embeddings and V4 auxiliary blocks are FP16.

generation:

10-11 tokens/s

User asks which model to use for agentic coding. User reports DeepSeek V4 Flash at 10-11 t/s with 1M context, and Qwen3.8 27B at 15-16 t/s with FP8 quants.

generation:

2.0 tokens/s

quant:

mixed 4/8-bit

User reports 2.04 t/s with an 8 GB expert cache and 1.23 t/s with a 32 GB expert cache, streaming from SSD. Speculative decoding and prefetch both decreased throughput. Codebook 2-bit quantization failed the quality gate.