Local model performance on your hardware

generation:

71.2 tokens/s

prompt processing (prefill):

1096.8 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Qwen3-Coder 30B-A3B at 71.2 t/s generation and 1096.8 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention enabled, run with -ngl 100 on one 60-core GPU. The figures are separate synthetic llama-bench tests: PP512 at 1096.84 +/- 5.48 t/s and TG128 at 71.23 +/- 0.08 t/s, with 16 CPU threads and a 16.45 GiB model footprint.

generation:

69.3 tokens/s

prompt processing (prefill):

1080.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

off

mtp (multi-token prediction):

off

User reports Qwen3-Coder 30B-A3B at 69.3 t/s generation and 1080.5 t/s prompt processing on a Mac Studio M2 Ultra with 128GB unified memory. Setup is llama.cpp build 6060 with the UD-Q4_K_XL GGUF and flash attention off, one GPU with -ngl 100 and 16 threads. The figures are separate synthetic llama-bench tests: PP512 and TG128, not a 512-input/128-output request. The same run measured 818.85 t/s PP8192 and 40.35 t/s TG8192.

generation:

26.4 tokens/s

prompt processing (prefill):

276.2 tokens/s

quant:

oQ8e mixed affine 8-bit + FP16 (MLX)

User reports Qwen3.8 27B at 26.4 t/s generation and 276.2 t/s prefill on an Apple Mac Studio M2 Ultra with 128 GB unified memory. Setup is oMLX 0.6.3 with oQ8e mixed affine 8-bit plus FP16 weights, run as an ANE and CPU split at a source-labelled 65,536-token prompt depth. The GPU-only control at the same depth is 231.2 prefill and 26.6 decode; the author reports gains apply to cold prefills, not prefix-cached turns, and peak memory at 64K rises from 44.0 GB to 60.0 GB with ANE.

generation:

23.0 tokens/s

prompt processing (prefill):

304.2 tokens/s

quant:

oQ4e (MLX)

User reports Qwen3.8 27B at 23 t/s generation and 304.2 t/s prompt processing on an M2 Ultra 128GB Mac Studio. Setup is oMLX 0.6.3 with oQ4e 4-bit quantization at workload depth 65536, with 16.7GiB of weights and peak memory rising from 25.3 to 36.2GB. The 304.2 PP is the ANE-on arm; the separate GPU-only baseline is 242.2 PP and 23.8 TG, and the source calls the decode change within variance. Depth 65536 is not a verified configured maximum context.