Local model performance on your hardware

generation:

20.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next-Heretic Q8_0 at 20.45 tok/s on an M5 Max 128GB. Setup is llama.cpp (closed PR #27739 fork) with Q8_0 and Q4_0 KV cache, -ngl 16, 4096 context, 45.1 GiB wired, MTP draft head enabled. Measured on 12 varied prompts with 24 warm-up queries discarded. The model is 122B total / 10B active MoE, 197 GB across 5 shards. A 262144-token server ran on llama.cpp v0.6.0 with ~32 tok/s prefill and decode falling to ~3.6 tok/s at 10k tokens. Under memory pressure a larger GPU tier wins: -ngl 24 at 10.29 tok/s with 15 GiB held by other work versus -ngl 6 at 5.75 tok/s with 45 GiB held. Contiguous layer placement beats scattered expert placement by 28.6%. MTP acceptance is 51-83%, helping +44% when experts are cached in RAM but costing 16-33% when they stream from SSD. Stock llama.cpp was 11-19% faster than the fork at 4k/64k/256k, and v0.6.0 another ~7% faster. No quality measurement was run for this file.

generation:

1.0 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi K3 (2.78T) at 0.96 tok/s on a MacBook Pro M5 Max 128GB, with the trunk in memory and 1.44 TB of 4-bit routed experts streamed from NVMe drives. Setup is a llama.cpp-derived fork with experts split across the internal SSD plus up to three Thunderbolt 5 enclosures, each holding a byte-identical replica. Decode scales with drive count: 0.55 tok/s on one drive, 0.75 with one enclosure, 0.89 with two, 0.96 with three (1.75x), and 1.00 steady over 512 generated tokens, 1.13 over 128.

generation:

105.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 105 tok/s decode at 128 context and 97 tok/s at 8k on an M5 Max 128 GB. Setup is llama.cpp with Q8_0 GGUF weights, MTP speculative decoding on, 65536 context, 45 GB memory. Without MTP the same model gives 93 tok/s at 128 and 87 tok/s at 8k; MLX gives 85 and 79 tok/s. MTP adds 75% on the dense 27B (17 to 32 tok/s) but only 12% on the MoE 35B-A3B. llama.cpp beats MLX by 10-24% on this machine. DeepSeek-V4-Flash under ds4 at q2-q4 (91 GB) reaches 33 tok/s at 128 and 28 tok/s at 8k.

generation:

24.0 tokens/s

prompt processing (prefill):

333.0 tokens/s

quant:

IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks GLM-5.3 321B across a MacBook M5 Max 128GB and an AMD Strix Halo 128GB connected over TB4. The default tensor split was slower than the Mac alone. A Mac-heavy split improved prompt processing from 188 to 333 t/s and generation from 17 to 24 t/s. User also reports 12-15 t/s for model sizes of 148-200 GB and plans to add two Asus GX10 next week.

generation:

79.4 tokens/s

prompt processing (prefill):

2153.0 tokens/s

quant:

Q8 XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks several models on an M5 Max 128GB and selects Qwen3.6 35B A3B as the primary model for its MoE detail. Other models tested include Gemma 4, Qwen 3.6 27B, MiniMax M2.7, Mistral Medium 3.5, Step 3.7 Flash and DeepSeek V4 Flash. Results show strong performance for MoE models.

generation:

31.8 tokens/s

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on an M5 Max 128GB across four engines. Setup is llama.cpp with native MTP at 31.8 t/s sustained and 22.7 t/s at 56k context, TTFT 125ms, prefix reuse 0.80, RAM 36.8GB; oMLX 0.6 at 29.8 t/s and 24.6 t/s at depth, TTFT 1968ms, prefix reuse 0.82, ~17GB; mlx-vlm with MTP at 43.3 t/s and 28.9 t/s at depth, TTFT 9982ms, no prefix reuse, 15.6GB; and MTPLX 2.6 at 27.1 t/s and 23.1 t/s at depth, TTFT 13400ms, no prefix reuse. The fastest engine, mlx-vlm, is unusable for agents due to 10s TTFT and no prefix reuse. Without --spec-type draft-mtp, llama.cpp drops the MTP head; a draft cap of 4 gives 31.8 t/s versus 21.9 t/s with no speculation, cap 2 from Qwen3.6 presets gives only 22.2 t/s, and cap 6 loses gains. At 262k context with q8_0 KV cache llama.cpp gives 29.6 t/s and 18.5 t/s at depth; at 131k with f16 it gives 31.8 t/s and 22.7 t/s at depth. The model does 5 searches then calls more tools instead of writing the brief and dies at max_turns. reasoning_effort 'none' throws an exception on 3.8, so enable_thinking: false is needed; oMLX does not activate MTP on 3.8; the model field of the response should be checked rather than /v1/models; and the M5 Max throttles to 50% after 80-110s. No generational verdict versus 3.6 is given.

generation:

11.5 tokens/s

prompt processing (prefill):

333.0 tokens/s

quant:

UD-Q2_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 79 GB in 2-bit running at 350K context for 3.5 hours on a 128 GB M5 Max, over 100 turns. Prefill runs 1,561 t/s at 5.6K context tapering to 318 t/s at 111K, with normal turns at 77-854 t/s from prefix reuse. Decode is about 30-35 t/s at small context down to 11.5 t/s at 169K, with a dip to 7.7 t/s at around 140K due to Low Power Mode. Decode figures are effective throughput with speculative decoding. The user calls the experience strong for the first ~100K context, then reports role confusion on long-tail tasks, suspecting the 2-bit quant or preview-model long-context quality.

generation:

33.9 tokens/s

quant:

Q4_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks prompt processing at 65,536 context, with throughput falling from 62.80 t/s at 128 context to 33.92 t/s at 65,536. Memory peaked at ~120+ GB, and the system stayed sluggish but usable.

generation:

32.4 tokens/s

quant:

UD128 (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Tencent HY3 295B-A21B MoE at 32.4 t/s generation at empty context and 16.3 t/s at 16K context on Metal. Setup is llama.cpp with the UD128 quant (107 GB), a q8_0 KV cache, and 24K context. With MTP n=2, generation peaks at 38 t/s. Prefill runs 528 t/s at pp512 empty and 124 t/s at pp512 at 16K. User compares the result favorably to DeepSeek V4 Flash.