Local model performance on your hardware

generation:

8.1 tokens/s

quant:

Q2_0 (GGUF)

User reports Qwen3.8-Flash-Next 125B at 8.1 t/s on an M4 Mac mini with 24GB unified memory. Setup is Strata with GSQ-RCO Q2_0 weights, Metal backend, 8K context configured and a 4GB expert weight cache, streaming expert weights from the SSD. The figure is the median of three 128-token generation runs with thinking disabled, temperature 0 and no MTP, on short prompts; the full 8K context was not tested. User calls the speed slow for everyday interactive work but usable for patient experiments, and notes frequent memory-pressure warnings with no swap use during the short tests.

generation:

22.6 tokens/s

prompt processing (prefill):

239.7 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

User reports Qwen 2.5 Coder 7B at 22.6 t/s generation and 239.7 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, full Metal GPU offload, 4 CPU threads, and F16 K/V cache. The figures are separate synthetic tests, PP512 and TG128, each averaged over three repetitions; the native rates were 239.73 +/- 2.07 t/s and 22.60 +/- 0.01 t/s.

generation:

46.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

User reports Llama 3.2 3B at 46 t/s generation and 530 t/s prompt processing on an M4 24GB Mac mini. Setup is llama.cpp llama-bench build 9100 with Q4_K_M, Metal backend, full GPU offload, F16 K/V cache, and four CPU threads. The figures are separate synthetic tests: PP512 at 530.01 +/- 0.86 t/s and TG128 at 45.95 +/- 0.02 t/s, three repetitions each, not a paired prompt-and-answer run.

generation:

6.5 tokens/s

prompt processing (prefill):

60.8 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

F16

mtp (multi-token prediction):

off

User reports Qwen3.8-27B at 6.5 t/s generation and 60.8 t/s prompt processing on a Mac mini M4 with 24 GB unified memory. Setup is llama.cpp build 9544 with the UD-IQ4_XS GGUF, run via llama-bench with -p 512 -n 128, F16 K/V cache and flash attention AUTO. The source's native figures are PP512 = 60.78 +/- 1.60 tokens/s and TG128 = 6.47 +/- 0.03 tokens/s, separate prompt and generation microbenchmarks rather than one request; TG128 is depth-0 generation, not decode after a 512-token prompt.

generation:

12.9 tokens/s

prompt processing (prefill):

138.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Gemma 4 12B at 12.9 t/s generation and 138.5 t/s prompt processing on a base Apple M4 Mac mini with 24 GiB unified memory. Setup is llama.cpp b10090 with UD-Q4_K_XL, run via llama-bench at 512 prompt and 128 generation tokens over 5 repetitions on one GPU. Later interleaved runs reached 12.71/12.79 t/s and prompt speed drifted 129–134 under sustained load; a depth ladder showed generation falling to 9.26 t/s at 32768.

generation:

17.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.5:9b at 17.42 tokens/s on an Apple M4 Mac mini with 24GB unified memory. Setup is Ollama 0.33.0 with Q4_K_M at 4096 tokens of context, single request, no KV-cache quantization reported. Generation speed is the median of 3 trials (range 17.33-17.57 t/s), measured with a ~88-token prompt and output capped at 300 tokens; model load time 2.3s, first-content latency 2.9s cold and 0.64s warm.

generation:

29.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E4B at approximately 29.5 tokens/s on a Mac mini M4 with 24GB unified memory. Setup is LM Studio 0.4.12 with llama.cpp engine 2.13.0 (Metal), GGUF Q4_K_M, 4096-token context, automatic GPU offload, temperature 0, 200 generated tokens per request; two trials gave 29.62 and 29.46 tokens/s with about 0.3s time to first token. A separate run at 32768 context and 12961 prompt tokens gave 25.95 tokens/s with 39.44s cold TTFT and 0.12s repeat TTFT from prefix caching. Four simultaneous requests with parallel=4 gave 9.66-9.69 tokens/s per request and 37.5 tokens/s aggregate. Source-reported measurements, not independently reproduced; prefill throughput, measured memory, KV-cache quantization, Flash Attention and speculative-decoding settings unspecified, and the 6.29GiB memory figure is an estimate.

generation:

11.5 tokens/s

quant:

4-bit (MLX)

User benchmarks Qwen 3 14B at 11.5 tok/s on a Mac Mini M4 with 24 GB unified memory. Setup is MLX with 4-bit quantization, using 8.4 GB peak memory and leaving 74% of system memory free with no swap. The study also reports Ollama q4_K_M results: Llama 3.2 3B at 42.0 tok/s, Gemma 2 9B at 15.2 tok/s, Qwen 2.5 14B at 10.8 tok/s, and DeepSeek-R1 32B at 0.28 tok/s. MLX Qwen 3 30B-A3B MoE reaches 44.8 tok/s but uses 16.3 GB and causes 2.74 GB swap; Qwen 3.5 35B-A3B crashes with Metal OOM at 20.4 GB. Tests ran under normal workstation load with ~12-13 GB already committed.