Local model performance on your hardware

generation:

28.8 tokens/s

prompt processing (prefill):

757.0 tokens/s

quant:

4-bit (MLX)

User benchmarks Nanbeige4.3-33B (dense, 4-bit) on a MacBook Pro M5 Max with 128 GB, comparing mlx-lm, llama.cpp and Kapitsa at prompt lengths from 512 to 32K tokens. At 512 tokens mlx-lm prefills at 757 t/s and decodes at 28.8 t/s, llama.cpp at 691 t/s and 27.3 t/s, and Kapitsa at 693 t/s and 34.9 t/s. mlx-lm and Kapitsa read the same 4-bit MLX AWQ pack (group 64); llama.cpp reads Q4_K_M. Kapitsa's decode figures use KFlash, its tree-based speculative decoding with a small drafter, while mlx-lm and llama.cpp run plain decoding. On a 512-token Python coding task Kapitsa reaches 36.6 tok/s greedy and 32.3 tok/s sampled, against 27.1 and 26.7 for mlx-lm and 24.9 and 24.8 for llama.cpp. At 29.5K context mlx-lm has the fastest cold TTFT at 59.4 s, Kapitsa the fastest follow-up TTFT at 300 ms and the lowest peak memory at 28.1 GB. The author works on Kapitsa and notes the KV cache is large, 26 GB for 128K tokens.

generation:

56.0 tokens/s

quant:

8-bit (MLX)

User reports Granite Switch 4.1 8B at 56 t/s chat on an M5 Max 128GB, with the 30B at 17 t/s. Setup is mlx-lm with a custom granite_switch package, 8-bit MLX weights, greedy decoding, one request at a time, medians of 3 runs. The 4-bit builds reach 98 t/s (8B) and 32 t/s (30B). Decision steps take 0.62 s on the 8B and 2.3 s on the 30B at a ~1.2K-token prompt. Accuracy on 500 labelled claims is 0.814 (8B 8-bit), 0.844 (30B 8-bit) and 0.840 (30B 4-bit), against 0.812 and 0.846 for the bf16 conversions. The user picks the 8B 8-bit for routing and classification.

generation:

101.0 tokens/s

quant:

8-bit (MLX)

User reports NVIDIA Nemotron-Labs Dory 1.0 4B at 101 tok/s on an M5 Max 128GB with one loop, dropping to 69 tok/s at 2 loops and 39 tok/s at 4 loops. Setup is an MLX port with the 8-bit 4.2 GB build, medians of repeated runs; peak memory is 7.7 GB at 32K context with one loop. After a 32K prompt the same runs give 82, 54 and 31 tok/s, with prefill at 3,200, 1,980 and 1,070 tok/s. Batch generation reaches about 450 tok/s total across 16 requests at 2 loops, and a 16 GB M1 Pro measured about 28 tok/s at 1K and 19 tok/s at 32K in 4-bit at 2 loops. 8-bit WikiText-2 perplexity falls from 12.18 to 11.77 to 11.70 as loops increase.

generation:

74.0 tokens/s

User compares Strata against oMLX running Qwen3.8-Flash-Next on a 128GB M5 Max, with Strata decoding at 74 tok/s versus 57.5 tok/s for oMLX at 245K context. Strata is a one-month-old engine built for this one model, running on Metal. The comparison holds one request at a time only, which the user notes as the current limitation.

generation:

140.0 tokens/s

quant:

4bit

User reports Qwen3.8-27B at 140 tok/s on an M5 Max MacBook Pro with 128GB, the fastest run of their open source Apple Silicon engine kapitsa. Setup is kapitsa with a 4bit model and kflash speculative decoding, where the 27B verifies a tree of drafter guesses in one pass rather than a single chain. The user is asking Mac Studio owners to run the same model for comparison.

generation:

50.3 tokens/s

prompt processing (prefill):

1084.0 tokens/s

quant:

oQ5e

User reports 50.3 t/s generation and 1084 t/s prompt processing. Source evidence (verbatim): Source: https://omlx.ai/benchmarks/performance/ogokptif # Qwen3.8-27B-oQ5e-mtp on M5 Max (40c) - oMLX Benchmark Hardware: Apple M5 Max (40-core GPU), 128 GB unified memory. Engine: oMLX v0.7.1.dev1. Model: Qwen3.8-27B-oQ5e-mtp, 5-bit. Multi-token prediction (MTP) is enabled with adaptive max depth 3, so this is a speculative decoding run; DFlash and SpecPrefill are disabled. Single request. At 8k context: 1,084 tokens/s prompt processing and 50.3 tokens/s token generation. At 4k context: 1,137 prompt and 58.4 generation; at 1k: 895.8 prompt and 54.6 generation; at 16k: 882.8 prompt and 75.5 generation.

generation:

154.0 tokens/s

quant:

oQ5e-mtp

kv:

bf16

User benchmarks Qwen3.6-35B-A3B at 154 t/s decode on a 1k prompt with oMLX's native engine on an M5 Max 128GB, falling to 55 t/s at 262k context. Setup is oMLX 0.7.1rc1 with the oQ5e-mtp quant, Lightning MTP on, TurboQuant KV off, and a 128-token greedy generation at each context step. Splash 1.3.2 with int8 KV decoded 147 t/s at 1k and 78 t/s at 262k, and used 30.2 GiB against native's 33.5 GiB prefill peak and 37.4 GiB decode. TurboQuant 5- and 6-bit decoded at 27-29 t/s at 262k, about half of off. The user also ran Qwen3.8-27B, where Splash int8 was faster at almost every context (69 vs 48 t/s at 1k) and native TurboQuant cost 7-25% decode speed with MTP on. No setting showed a detectable accuracy loss of more than about 2 points.

generation:

3.6 tokens/s

quant:

INT4 (GGUF)

User reports running DeepSeek v3.1 671B Terminus at 3.6 tokens/sec on a MacBook Pro with M5 Max 128GB and 8TB storage. Setup uses a heavily modified slipstream llama.cpp base that caches MoE experts to disk and calls them in as they change, with a -0.1 bias in the MTP router for in-RAM experts. A custom GGUF-derived format uses BF16 for fixed layers and INT4 for expert layers. The user says it is working extremely well and generating a somewhat usable 3.6 tokens/sec.

generation:

198.6 tokens/s

prompt processing (prefill):

3657.8 tokens/s

quant:

oQ5e

User benchmarks Qwen3.6-35B-A3B at 198.6 tok/s generation and 3657.8 tok/s prompt processing on an M5 Max 128GB. Setup is oMLX 0.7.1dev2 with oQ5e quant, Lightning MTP speculative decoding, and TurboQuant off, at 1024-token prompt and 128-token generation. Splash engine reaches 182.0 tok/s generation and 6040.8 tok/s prompt processing on the same test; continuous batching at 4x reaches 275.2 tok/s aggregate generation. A second model, Qwen3.7-27B, is also benchmarked.

generation:

79.1 tokens/s

quant:

MXFP4 (MXFP4)

User reports 79.1 t/s generation. Source evidence (verbatim): Source: https://bestllmfor.com/guides/macbook-pro-m5-max-128gb-ai-benchmark/ # MacBook Pro M5 Max 128GB AI benchmark Joel Ramat ran these tests on his own 16-inch MacBook Pro on September 16, 2026. M5 Max, 18 CPU cores, 40 GPU cores, 128 GB unified memory advertised at 614 GB/s, macOS 27.0, Ollama 0.34.1, plugged in, default GPU memory limit. All figures are raw outputs from Ollama. Each model was run twice with the same prompt (a 300 word explanation of unified memory versus VRAM) and only the second pass kept, one model loaded at a time. Generation speed: gpt-oss 120B MXFP4 (65 GB) 79.1 tok/s; gpt-oss 20B MXFP4 (13 GB) 113.4 tok/s; Qwen 3.6 35B-A3B MLX (23 GB) 167.2 tok/s; Qwen 3.8 27B MLX (18 GB) 68.0 tok/s; Qwen 3.8 27B GGUF Q4 (17 GB) 36.6 tok/s; Gemma 4 12B GGUF Q4 (7.6 GB) 58.1 tok/s. Prefill on a 16,689 token document with gpt-oss 120B: 1,388 tok/s, time to first token 12.0 s, generation 67.6 tok/s on that run. Only short bursts were tested, one machine, one operator.

generation:

8.4 tokens/s

quant:

oQ4e

mtp (multi-token prediction):

on

User reports Qwen3.8 oQ4e-mtp restored-prefix B3 / PR1 61bd8a06 at 8.4 t/s on an M5 Max 128GB. Setup is oMLX with oQ4e 4-bit weights and TurboQuant 4-bit KV cache, three greedy requests batched at roughly 20k / 45k / 135k prompt tokens with SSD prefix cache enabled. The 8.4 t/s is the per-row rate at the 20k row, measured at 8.35 t/s; the other rows ran 8.83 and 7.84 t/s per row, and the batch phase took 66 seconds.

generation:

30.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at ~30 tok/s decode at 2K context on an M5 Max 128GB, streaming a 341 GB Q2 GGUF from SSD. Setup is a fork of antirez's ds4 engine (sf-ds4-1flash) with Metal backend, one machine, weights streamed from disk. Decode is ~26 tok/s at 32K context and settles around 24 tok/s after an hour of heavy use. First token after an 8K prompt dropped from 1.23 s to 0.15 s. Prefill improved 45-99% and decode 62-78% versus upstream ds4. Output is bit-identical to ds4.

generation:

144.4 tokens/s

quant:

MLX 4-bit affine, group 64 (MLX)

User reports Qwen3 30B-A3B at 144.4 t/s on an M5 Max with 128GB unified memory. Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1 with a 944-token prompt and 512 output tokens. The figure is the arithmetic mean of six fresh-process runs, ranging 144.10 to 145.01 t/s; an altered 6-expert routing gave 155.61 t/s but is not a standard-model result.

generation:

36.1 tokens/s

prompt processing (prefill):

498.3 tokens/s

quant:

2.44bit-mixed (MLX)

User reports DeepSeek V4 Flash 0731 at 36.1 t/s generation and 498.3 t/s prefill on a MacBook Pro M5 Max 128GB 40 GPU. Setup is oMLX 0.5.7 with a 2.44 bpw mixed MLX quantization (92.8 GB on disk) at 1k prompt, single request, 128 generated tokens, speculative decoding off. Generation holds 33.8 t/s at 4k and 31.5 t/s at 32k, with peak memory rising 1.03 GB across an 8x context jump; prefill at 32k takes 97 seconds to first token. Continuous batching reaches 83.2 t/s at batch 8 (2.30x).

generation:

205.0 tokens/s

quant:

4bit (MLX)

User benchmarks Gemma 4 E2B at 205 t/s decode at 4k context on an Apple M5 Max with 128 GB unified memory. Setup is mlx-vlm 0.4.4 with MLX 0.31.1, 4bit weights, and TurboQuant 2.5-bit KV cache quantization; the model uses 4.7 GB at 4k. Decode falls to 78 t/s at 256k context. TurboQuant gives +15-19% at 128-256k on larger models but is negative on E2B. The same report also covers Gemma 4 E4B, Gemma 4 26B-A4B, Gemma 4 31B, Qwen 3.5 2B and Qwen 3.5 4B.

generation:

20.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

Q4_0

User reports Qwen3.8-Flash-Next-Heretic Q8_0 at 20.45 tok/s on an M5 Max 128GB. Setup is llama.cpp (closed PR #27739 fork) with Q8_0 and Q4_0 KV cache, -ngl 16, 4096 context, 45.1 GiB wired, MTP draft head enabled. Measured on 12 varied prompts with 24 warm-up queries discarded. The model is 122B total / 10B active MoE, 197 GB across 5 shards. A 262144-token server ran on llama.cpp v0.6.0 with ~32 tok/s prefill and decode falling to ~3.6 tok/s at 10k tokens. Under memory pressure a larger GPU tier wins: -ngl 24 at 10.29 tok/s with 15 GiB held by other work versus -ngl 6 at 5.75 tok/s with 45 GiB held. Contiguous layer placement beats scattered expert placement by 28.6%. MTP acceptance is 51-83%, helping +44% when experts are cached in RAM but costing 16-33% when they stream from SSD. Stock llama.cpp was 11-19% faster than the fork at 4k/64k/256k, and v0.6.0 another ~7% faster. No quality measurement was run for this file.

generation:

70.0 tokens/s

quant:

8-bit

kv:

INT8

User reports Qwen3.8-27B at 70+ tok/s generation on an M5 Max 128GB MacBook Pro. Setup is the MLTF engine with 8-bit weights and an INT8 KV cache, using Metal TensorOps W8A8 kernels. Cold prefill is 1,000+ tok/s on a 2Ki prompt, and aggregate generation throughput is 200+ tok/s at concurrency 4. The user chose 8-bit over 4-bit for quality.

generation:

86.7 tokens/s

quant:

Q2 (GGUF)

User reports Qwen3.8 Flash-Next Q2 at 86.7 tok/s with MTP on an M5 Max 128GB, using sf-q3-8flash, their stripped-down fork of antirez's ds4 engine. Setup is the sf-q3-8flash fork of ds4 on Metal with Q2 weights, MTP speculative decoding on, the whole model in unified memory. Stock ds4 ran the same Q2 MTP setup at 75.8 tok/s; on Q4 MTP went from 77.8 to 85.9 tok/s. No plain decode figure is given, only a 9-13% decode gain on Q2 and 2-11% prefill gain on Q4. Output stays bit-exact against stock ds4. An optional SSD streaming mode for experts on a simulated 48GB machine ran Q2 at 27 tok/s, around 35 tok/s with MTP.

generation:

130.6 tokens/s

prompt processing (prefill):

5402.4 tokens/s

User reports Qwen3 30B-A3B at 130.628 t/s generation and 5402.434 t/s prompt processing on an Apple M5 Max with 128GB unified memory. Setup is MLX 0.32.2 with MLX-LM 0.31.3 on macOS 27.0; quantization, context length and KV cache settings are not reported. Figures come from a third-party hardware review (Tbreak, September 21, 2026), not a reproducible benchmark log; prompt and generated token counts are unknown.

generation:

24.4 tokens/s

prompt processing (prefill):

636.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at Q2 running on an M5 Max 128GB Mac, with a fork of ds4 reaching 24.4 t/s generation at 2048 context versus 12.1 t/s for stock ds4. Setup is ds4 with a 341 GB GGUF (152 GB weights plus 189 GB Engram tables) streamed from SSD, no KV quant, bit-exact output. The fork doubles steady decode to 25.7 t/s at 2048 context and 22.0 t/s at 32768 context, and prefill from 16k to 32k context rises from 404 t/s to 636 t/s. Adding a byte-identical external Thunderbolt 5 SSD copy cuts time to first token on a 3.5K prompt from 13.2 s to 11.2 s and after an 8K context from 1.38 s to 0.18 s; decode is unchanged. The enclosure runs at PCIe 4.0 x4, about half the internal SSD speed. A two-SSD attempt in upstream ds4's mmap path was bit-exact but 13-38% slower on prefill and was not submitted.