Local model performance on your hardware

generation:

152.5 tokens/s

quant:

AXQ MXFP4 mixed (MLX)

mtp (multi-token prediction):

on

User reports Cyber-Tiel-Coder-35B-A3B at 152.5 t/s on an M2 Ultra 192GB Mac Studio. Setup is the AX Engine with an AXQ MXFP4 trunk and protected MTP sidecar, one Metal GPU, expert streaming off. The figure is the median decode rate over six samples on a 219-input/256-output python-lru workload, with TTFT 505.93 ms; completion including TTFT was 117.59 t/s.

generation:

134.6 tokens/s

quant:

AXQ-MXFP4 (MLX)

mtp (multi-token prediction):

on

User reports Tiel-Coder 35B-A3B at 134.6 t/s median native decode on a single Apple Mac Studio M2 Ultra with 192 GiB unified memory. Setup is AX Engine with the AXQ-MXFP4 MLX export, explicit throughput-MTP at maximum draft depth three, expert streaming off, and the model kept fully resident. Separate median completion throughput is 107.28 t/s including TTFT, with median native TTFT 494.41 ms; the 256-token output is total output, not the decode numerator.

generation:

22.2 tokens/s

prompt processing (prefill):

325.3 tokens/s

quant:

MXFP4 / affine 8-bit mixed (MLX)

mtp (multi-token prediction):

on

User reports Qwen3.8 Flash-Next at 22.2 t/s generation and 325.3 t/s prefill on an M2 Ultra 192GB. Setup is AX Engine at MXFP4 / affine 8-bit mixed, speculative decoding on, one request. Medians of 20 trials in defai-digital's AutomatosX benchmark of 2026-10-08. A source-reported measurement, not independently reproduced.

generation:

32.7 tokens/s

prompt processing (prefill):

420.8 tokens/s

quant:

UD-Q4_K_M (GGUF)

kv:

Q8

flash attention:

on

User reports MiniMax M2.7 at 32.7 t/s generation and 420.8 t/s prefill on an M2 Ultra 192GB. Setup is llama.cpp at UD-Q4_K_M, one request, speculative decoding off. Token-weighted rates across four sequential agentic tasks in crypdick's run of 2026-08-10. A source-reported measurement, not independently reproduced.

generation:

37.9 tokens/s

prompt processing (prefill):

301.6 tokens/s

quant:

UD-Q3_K_M (GGUF)

kv:

q8_0

User reports MiMo-V2.5 at 37.87 t/s generation and 301.61 t/s prompt processing on an M2 Ultra 192GB Mac Studio. Setup is llama.cpp llama-server with UD-Q3_K_M GGUF, q8_0 KV cache, 65,536 context, Flash Attention on, gpu_layers 999, batch 2048, and 8,192 MiB cache_ram. Figures are token/time-weighted across four sequential real-world agentic tasks (n=1 each), not llama-bench or end-to-end throughput; one task hit its 8,192-token output cap and 258 of 2,290 prompt tokens were cached. Server peak RSS was 131.015 GiB.

generation:

7.7 tokens/s

quant:

8 bit (GGUF)

User compares GGUF and MLX inference speeds on a Mac Studio M2 Ultra with 192GB, using LM Studio 0.3.10 with MLX 0.6. For Qwen2.5 72B 8-bit, GGUF reaches 7.68 tokens/sec while MLX drops to 0.42 tokens/sec. Smaller models show the opposite: Qwen2.5 0.5B 4-bit runs at 79 tokens/sec in GGUF versus 317 tokens/sec in MLX, and Mistral Small 22B 8-bit is roughly equal at 25.01 versus 25.96 tokens/sec. User reports MLX is 4-5x faster for very small models but about 10x slower for large ones, with the inflection point around 22B parameters, and notes MLX does not use much GPU for larger models while LM Studio lacks a layer-offload option for MLX.

generation:

22.4 tokens/s

prompt processing (prefill):

360.2 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

Q8

User benchmarks Qwen3.8-27B at 22.38 t/s generation (tg128) and 360.22 t/s prompt processing (pp512) on a Mac Studio M2 Ultra with 192GB unified memory. Setup is llama.cpp build b10450 llama-bench with the Unsloth UD-Q6_K_XL quant (24.13 GiB), flash attention on, q8_0 K and V cache and batch and micro-batch 2048. The user serves the model through llama-server at 131072 context with MTP speculative decoding (draft-mtp, draft max 3), and gets around 16 t/s on the WebUI with a flappy bird game prompt.

generation:

25.8 tokens/s

prompt processing (prefill):

250.0 tokens/s

User reports DeepSeek V4 Flash at 25.8 t/s, with a 42 t/s peak, on an M2 Ultra with 60 cores and 192gb. Setup is a custom llama.cpp fork with the model repacked losslessly to 141GiB, smaller than the public GGUFs, no KV cache quant, and an SSD KV cache with dynamic lanes, 1M context total across 8 lanes. Prompt processing is about 250 t/s at 8k-32k. The user compares this with 16 t/s on an M3 Ultra.

generation:

28.0 tokens/s

User reports decode speeds at various context depths: 28 t/s at the start, 23.5 t/s at 45k, and 18 t/s at 192k. The run was maintained with 8k token output. Prefill performance is mentioned but no numbers are given.