Local model performance on your hardware

generation:

64.4 tokens/s

prompt processing (prefill):

645.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder 30B-A3B at 64.4 t/s generation and 645 t/s prompt processing on an M1 Max 64GB. Setup is llama.cpp with Q4_K_M, measured via llama-bench on Metal. The figure is a native prefill/decode rate, not a TTFT-derived estimate; exact test lengths, context, batch settings and KV precision are not given.

generation:

0.9 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports IQ2_XS quantized 7B model at 0.95 t/s on an M1 Max 64GB, with 25 tokens generated in 26.30 seconds at context 1523. Setup is llama.cpp with IQ2_XS quantization; the user also mentions running Mixtral 8x7B fine-tunes at Q5_K_M getting 8-15 t/s. User also reports an IQ2_XS 120B model taking about 9 minutes for prompt evaluation and roughly 0.1 t/s generation, and questions whether the slow IQ quant performance on Apple Silicon is expected or a bug.

generation:

8.0 tokens/s

prompt processing (prefill):

30.0 tokens/s

quant:

IQ3-XXS

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at about 8 t/s decode and about 30 t/s prefill on an M1 Max 64GB. Setup is a patched llama.cpp with the IQ3-XXS quant at 104 GB and context limited to 64k.

prompt processing (prefill):

180.0 tokens/s

quant:

Q4 (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a custom Q4 quant with spliced tensors from Unsloth and AtomicChat quants, with MTP enabled giving +70% decode at 22 btps. Setup uses custom metal-optimized sparse attention, with prefill reduced to 170 tps at 4K and 150 tps at 256K context. Q4_0 MTP matches Q8_0 acceptance rates at half RAM.