M4 16GB reports

generation:

45.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 45.4 t/s generation on a Mac mini M4 16GB. Setup is Ollama 0.30.8 at Q4_K_M on Metal, one request, speculative decoding off. Mean of two runs by ai-local-lab, measured 2026-06-16. A source-reported measurement, not independently reproduced.

generation:

30.9 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a ~3.8x generation speed regression on Apple M4 Mac Mini (16 GB unified memory) with Gemma 4 26B A4B IQ4_XS. Master build (d12cc3d) achieves 8.14 tok/s versus 27.18 tok/s on tag b8680, both built from source with llama-server and identical flags. Setup uses Metal backend, --flash-attn on, q4_0 KV cache, 65536 context, and -ngl 99. The regression is reproducible and not explained by memory or swap pressure; prefill on b8680 was 55.7 tok/s.

generation:

45.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 E2B at ~45 tokens/s single-task on three M4 Mac minis with 16GB RAM, each handling around 10 parallel tasks at ~6 tokens/s each. Setup is local inference on Apple Silicon; no engine, quant, or context length stated. The user asks whether one Mac Studio or more Mac minis would give better throughput for parallel labeling and categorization.

generation:

25.6-28.3 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 E4B at 25.6–28.3 tok/s on an M4 alone, with TTFT of about 0.31 s on a 71-token prompt. Setup is Ollama with 4-bit weights; a distributed run across the M4 and an RTX 3060 6 GB over Wi-Fi reached about 8 tok/s with layers 0–14 on the M4 and 14–42 on the RTX, and about 10.2 tok/s with layers 0–4 on the M4 and 4–42 on the RTX. Gemma 4 E2B runs at 42.8 tok/s on the M4 alone, up from about 11.5 tok/s before optimising the decode path. The user notes that adding a second machine does not automatically speed up inference and is exploring task-level rather than layer-level distribution.

generation:

25.0 tokens/s

quant:

4-bit (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 25 tokens/s on short prompts running a 35B MoE model at 4-bit with 131K context on two base M4 Mac minis (16 GB each) over Thunderbolt 4 using llama.cpp RPC mode. Setup splits the model across the two M4s with a 50/50 memory split; a 40/60 split OOMed one machine. An M1 Mac mini acts as gateway with an OpenAI-compatible API and HTTP proxy. A 64K-token prompt takes almost five minutes to first token, and prompt cache hits arrive roughly 70x sooner. User notes reasoning was on by default and that the setup is good for background agents and learning but cloud models are still used for daily work.

generation:

20.0 tokens/s

prompt processing (prefill):

40.0 tokens/s

quant:

ternary

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Maple Preview, a 20B A1B MoE trained from scratch in ternary precision, generating at around 20 tokens/s on a MacBook Air M4. Setup is the user's Maple integration in Mference, a fork of turbo-fieldfare that keeps the weights on SSD and streams MoE experts from disk, using between 500 and 1,200 MB of memory depending on context length. Prompt processing runs at around 40 tokens/s, although the title gives 40 tps generation. On Tau-2 the user measured 0.48 Airline, 0.175 Retail and 0.427 Telecom.