Local model performance on your hardware

generation:

23.0 tokens/s

prompt processing (prefill):

29.6 tokens/s

quant:

UD-Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 23.0 tok/s generation and 29.6 tok/s prompt reading on a single prompt, on a 16 GB Mac mini M4. Setup is stock llama.cpp b11715 with the Kraits plugin, Unsloth UD-Q4_K_M (22.1 GB) weights, 16K context, temperature 0 and thinking off. A 4-agent crew writing a TypeScript LRU cache ran at 16.0-23.8 tok/s writing and 37.8-47.3 tok/s reading at ~16K context, scoring 9/9; the ~8K run scored 8/8 and the ~1K run 6/8. The single prompt scored 8/8 in 43.7 s.

generation:

38.0 tokens/s

prompt processing (prefill):

274.0 tokens/s

quant:

UD-IQ2_M (GGUF)

User reports Qwen3.6-35B-A3B at 38.0 tok/s on a Mac mini M4 16GB, running a four-agent CrewAI coding crew (analyst, architect, developer, tester) via Coachwhip v0.5.3. Setup is Coachwhip with UD-IQ2_M quant (11.5 GB) and all experts in GPU memory; temperature 0.7, top-p 0.8. The developer agent wrote 1,069 tokens at 38.0 tok/s and read 810 tokens at 274 tok/s; the tester wrote 1,811 tokens at 36.4 tok/s and read 2,071 at 281 tok/s. The crew scored 8/8 on tests written by gpt-5-mini, matching the OpenAI gpt-5-mini run (8/8) at 2 min 19 s versus 1 min 41 s. Two other runs scored 7/8, both missing fractional refill. Prompt reading was about 300 tok/s on a 1,500-token prompt, 242 at 7.5K, 227 at 9.3K, and 202 at 16.8K.

generation:

60.0 tokens/s

quant:

Q8

User reports JetBrains Mellum 2.1 12B A2.5B MoE running at around 60 t/s on an M4 16GB at full 130k context. Setup uses a Q8 quant; the user notes a 16 GB GPU can run it easily. The user also got 140 t/s on a 9070 XT and expects 8GB and 12GB cards to reach 30-40 t/s. They find it codes well but suggest a bigger model for detailed planning.

generation:

90.1 tokens/s

prompt processing (prefill):

2122.0 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-0.8B at 90.1 t/s generation and 2122 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.6.0 build 11429 with Q8_0 and Metal, flash attention enabled, single-stream. The same campaign's Homebrew llama.cpp 0.4.1 build 10964 baseline reached 87.3 t/s generation and 2049 t/s prompt processing; batched aggregate generation was 87.7 t/s at 1 stream and 322.4 t/s at 8 streams.

generation:

28.3 tokens/s

prompt processing (prefill):

352.4 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.5-4B at 28.3 t/s generation and 352.4 t/s prompt processing on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp 0.6.0 build 11429 with Metal, Q4_K_M GGUF, flash attention enabled, run via llama-bench with -p 512 -n 64 -d 0,4096 -r 3. The two figures are separate microbenchmarks, not one 512-input/64-output request; generation context depth is unknown, and PP512 after 4096 existing tokens is 321.5 t/s.

generation:

47.4 tokens/s

prompt processing (prefill):

556.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen2.5 3B at 47.4 t/s generation and 556.9 t/s prompt processing on a base Apple M4 Mac mini with 16 GB unified memory. Setup is llama.cpp with Q4_K_M, all layers on the GPU via Metal, measured with llama-bench PP512 and TG128 passes. The figures are means of three repetitions, with 10.42 GiB baseline system RAM in use during the run, so it is not an idle-machine result.

generation:

13.1 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 12B at 13.1 t/s on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp with Q4_K_M and 8,192 tokens of context, run on the official Mac build on Apple graphics. The 13.1 t/s is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; the source omits measured Mac memory.

generation:

17.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.5 9B at 17.7 t/s on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp with Q4_K_M and a configured context of 8,192 tokens. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; no prompt-processing rate, TTFT or end-to-end latency is supplied.

generation:

29.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.5 4B at 29.7 t/s on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp with Q4_K_M and a configured context of 8,192 tokens; KV cache precision is not stated. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; no prompt-processing rate or TTFT is given.

generation:

5.8 tokens/s

prompt processing (prefill):

60.5 tokens/s

quant:

UD-Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 5.8 t/s generation and 60.5 t/s prompt processing on two Apple Mac mini M4 machines with 16 GB unified RAM each. Setup is llama.cpp build 9731ad3 with Metal and RPC, Unsloth UD-Q4_K_M, model split 50/50 over a Thunderbolt 4 bridge, with -fa 1 -ngl 99 -b 1024 -ub 512 and three repetitions. The 5.8 t/s is the tg128 native llama-bench microtest and 60.5 t/s is the separate pp512 phase; one mini cannot fit the 16.5 GB artifact at its 13 GB GPU limit.

generation:

12.7 tokens/s

prompt processing (prefill):

116.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3-14B-4bit at 12.7 t/s generation on one Apple Mac mini M4 with 16 GB unified memory. Setup is mlx-lm 0.29.1 native mlx_lm generate with 4-bit MLX safetensors, --temp 0 and --max-tokens 128, no speculative decoding or prompt cache. The separate long run reports 116 t/s prompt processing at 3,796 input tokens; the two short runs both measured 12.7 t/s, and peak memory was 8.4GB.

generation:

23.2 tokens/s

prompt processing (prefill):

219.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3-8B-4bit at 23.2 t/s generation on a single Apple Mac mini M4 with 16 GB unified memory. Setup is MLX via mlx-lm 0.29.1 native mlx_lm generate, 4-bit, with --temp 0 and --max-tokens 128; no speculative decoding or prompt cache. A separate long request reports 219 t/s prompt processing at 3,796 input tokens, so the two figures come from different requests. Peak memory is reported at 4.7GB.

generation:

17.0 tokens/s

User reports MiniCPM-V 4.7 35B-A3B at 17+ tok/s on an iPad Pro M4 16GB, with 13+ tok/s on an iPhone 17 Pro Max and about 3GB memory usage. Setup details are not stated; the post highlights the model's LES (Lookahead Expert Scheduling) technique for MoE models, claimed to give up to 5x speedup with identical outputs. The post also cites 90.4 OCRBench with 16x visual compression, 34 multimodal benchmarks, a new 1B lightweight VLM, Self-Replay Fine-Tuning, and support for Apple, Intel, AMD, Arm and Rockchip.

generation:

64.0 tokens/s

quant:

NVFP4 (MLX)

User reports Mellum2.1 at 64 tokens/s at 1K context on an M4 16GB Mac. Setup is MLX with an NVFP4 conversion, 6.84 GB on disk. The user converted JetBrains' Mellum2.1 to MLX NVFP4 and asks for feedback on real tasks.

generation:

46.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.2 3B at 46.7 t/s on an M4 16GB Mac mini running macOS 15.3.1. Setup is Ollama 0.31.2 with llama.cpp and Metal, Q4_K_M, default settings, fresh runtime launch. The figure is the median of three requests after one discarded warm-up, with a source-reported max-minus-min spread of 0.1 t/s; configured context, KV precision and speculative decoding are unknown.

generation:

21.0 tokens/s

prompt processing (prefill):

229.2 tokens/s

quant:

Q4_K_M (GGUF)

User reports Llama 3.1 8B at 21 t/s generation and 229.2 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.4.0 with Q4_K_M and Metal, all layers on GPU; context, KV precision and Flash Attention are unknown. The figures come from separate llama-bench tests, pp512 at 229.23 ±0.08 t/s and tg128 at 20.95 ±0.02 t/s over three runs, not a combined 512-input/128-output completion, with 10.29 GiB RAM already in use.

generation:

8.2 tokens/s

prompt processing (prefill):

9.1 tokens/s

quant:

EVR-1 custom compression (~3-bit weights) (GGUF)

User reports Llama 3.1 8B at 8.2 t/s generation and 9.1 t/s prompt processing on an M4 16GB Mac mini. Setup is Evrmind's custom Metal binary with EVR-1 compression at roughly 3 bits per weight and full GPU offload. The figures come from separate pp512 and tg128 phase tests, not a paired 512-input/128-output run; the source table lists 9.11 and 8.16 t/s.

generation:

16.5 tokens/s

prompt processing (prefill):

518.3 tokens/s

quant:

dynamic_wi8_afp32

User reports Gemma 4 at 16.5 t/s generation and 518.3 t/s prompt processing on an M4 with 16 GB unified memory. Setup is LiteRT-LM on the GPU backend with dynamic_wi8_afp32 quantization and a 4,096-token maximum context, using a 256-token prefill and 256-token decode benchmark. The source gives exact rates of 16.53 t/s decode and 518.28 t/s prefill, and does not establish whether the phases were separate tests or one request.

generation:

3.6 tokens/s

prompt processing (prefill):

21.0 tokens/s

quant:

TurboQuant 3-bit g64 + 4-bit affine extras (MLX)

User reports Qwen3.8 27B at 3.6 t/s generation and 21 t/s prefill on an M4 Mac mini with 16 GB unified memory. Setup is turboquant-mlx 0.24.0 with TurboQuant 3-bit g64 weights and 4-bit affine embeddings, lm_head and vision tower, at a 2014-token prompt with 13.14 GiB measured peak. The source's four rows span 220 to 5017 prompt tokens, with decode 3.6 to 3.9 t/s and prefill 19.0 to 21.0 t/s; generated-token counts and KV precision are unstated.

generation:

10.0 tokens/s

quant:

4bit (MLX)

User reports Qwen3-8B at ~10 tok/s decode on a 16GB Mac (M4) using MLX with 4-bit quantization. Setup runs a persistent local agent with Qwen3-8B on the GPU, Qwen3-4B as 36 int8 layer shards on the Neural Engine, and Qwen2.5-0.5B for routing. The ANE toolchain is custom Rust with no Python. User notes 16GB is a hard ceiling and the mesh has only run on one machine so far.