Local model performance on your hardware

generation:

41.4 tokens/s

prompt processing (prefill):

198.2 tokens/s

quant:

4-bit dynamic/GS32 (MLX)

User benchmarks Qwen3.8-27B at 41.4 tok/s decode and 198.2 tok/s prefill at 1K context on an Apple M3 Max 128GB (40-core GPU). Setup is MTPLX v2.8.2 with a 4-bit dynamic/GS32 MLX model, MTP depth 3, single request, 128-token generations, 3-minute GPU rests between rounds. MTPLX decode falls to 19.8 tok/s and prefill to 115.5 tok/s at 16K, where oMLX oQ4e reaches 27.0 tok/s decode and 231.7 tok/s prefill; the 8-bit MTPLX Quality build is nearly identical in speed (18.8 tok/s, 121.3 tok/s) with KL 0.00105 versus Speed's 0.0220. ANE prefill on oMLX gains 57.0% at 32K (126.1 to 198.1 tok/s).

generation:

90.8 tokens/s

quant:

4-bit (MLX)

User benchmarks Qwen3.5-35B-A3B (thinking disabled) on Apple Silicon, reporting effective throughput of 71.3 tok/s on the ops-agent scenario with an M3 Max 128GB (40 GPU) running oMLX at MLX 4-bit; generation speed was 90.8 tok/s. Setup is oMLX with a tiered KV cache; the same M3 Max run gives 61.4 effective tok/s on doc-summary, 22.6 on prefill-test and 90.1 on creative-writing. Generation speed is similar across MLX engines (~55-93 tok/s depending on hardware) but prefill varies widely: at 8K context LM Studio MLX takes 49s to prefill while oMLX takes 1.7s with its persistent SSD cache from a prior run. Effective tok/s is output tokens divided by total wall-clock time including prefill. On the same M3 Max, LM Studio MLX reaches 37.1 effective tok/s; on M1 Max 64GB (24 GPU) oMLX 4-bit reaches 37.5 and oMLX fp16 47.3.

generation:

98.8 tokens/s

prompt processing (prefill):

2358.0 tokens/s

quant:

4-bit

User benchmarks Qwen3.6-35B-A3B at 98.8 tok/s decode and 2,358 tok/s prefill on an M3 Max 128GB using Splash 1.0.1 with 4-bit weights and DFlash speculative decoding. Setup is Splash 1.0.1 on macOS 26.6.2, 4-bit proprietary format, ~22K context, everything resident on GPU. Warm TTFT at 22K is 98 ms; 4 parallel streams reach 181 tok/s aggregate. User compares against Ollama NVFP4 (MLX) at 84.7 tok/s decode and oMLX serving Ornith 1.5 at 96.2 tok/s, and notes DFlash gains 2.74x on predictable text (239 tok/s) versus 87.2 tok/s on technical prose. User also benchmarks Qwen3.8-27B dense at 30.0 tok/s decode and 292 tok/s prefill, below Ollama's MLX engine at 42.5 tok/s.

generation:

31.7-35.2 tokens/s

quant:

IQ3_XXS (GGUF)

User reports a Qwen3.8-Flash-Next MTP draft head for GGUFs on llama.cpp, measured on an M3 Max 128GB with Metal. Setup is llama.cpp with an IQ3_XXS trunk and an unsloth shared Q8_0 MTP head, one session. Decode is 31.7-35.2 tok/s with MTP against 23.8-27.4 without, draft acceptance 84-99% at 2 drafts. Five concurrent agent sessions cost 8-20% throughput at 2 drafts, while 1 draft at 89% acceptance is neutral. The work is blocked on unmerged llama.cpp PR #28243.

quant:

4bit

User runs Qwen3.6-35B-A3B-4bit on an M3 Max 128GB for production sub-agent delegations. User also mentions GLM-5.1 for orchestration. User is considering building a 5090 rig.

generation:

5-6 tokens/s

prompt processing (prefill):

160.0 tokens/s

quant:

Q8 (MLX)

User reports 160 t/s prefill and 5-6 t/s generation on an M5 Max 128GB with Qwen 3.6 27B Q8 MLX at 290k context. GPU utilization sits at only 36-50%. User expected 8-14 t/s generation and asks how other setups compare.