Local model performance on your hardware

generation:

73.4 tokens/s

prompt processing (prefill):

1808.0 tokens/s

quant:

Q8_0 (GGUF)

kv:

f16

User reports Qwen3.5-35B-A3B at 73.43 tokens/s generation and 1808 tokens/s prefill on an Apple M3 Ultra 96GB. Setup is llama.cpp build 8860 with Q8_0 GGUF weights and f16 KV cache, Metal backend, single machine. The figures are separate synthetic tests (TG128 and PP512), not an end-to-end request. Historical TG128 85.48, turbo3 KV 63.23 and PP8192 1183 belong to other configurations and are kept for comparison only.

generation:

73.0 tokens/s

prompt processing (prefill):

347.0 tokens/s

quant:

MXFP4 (MLX)

User reports 73 t/s generation and 347 t/s prompt processing on a Mac Studio M3 Ultra with 96GB, running mlx_lm.server with no speculative decoding and one request at a time. Source evidence (verbatim): Source: https://elitetestengineering.substack.com/p/gpt-oss-120b-on-two-generations-of # GPT-OSS-120B on Two Generations of Mac Studio: The Long-Context Gap Gets Bigger, Not Smaller Two Mac Studios running gpt-oss-120b (mlx-community/gpt-oss-120b-mxfp4-bf16, about 65GB) through mlx_lm.server. M5 Ultra: 30 CPU cores, 64-core GPU, 256GB unified memory. M3 Ultra: 28 CPU cores, 60-core GPU, 96GB unified memory. Short prompt of about 146 tokens: M5 Ultra prefill about 555 tok/s and decode 99 tok/s, M3 Ultra prefill about 347 tok/s and decode 73 tok/s. Long prompt of about 24,400 tokens: M5 Ultra prefill about 3,955 tok/s and decode 76 tok/s, M3 Ultra prefill about 1,101 tok/s and decode 55 tok/s. No speculative decoding was used. The M5 Ultra ran with decode concurrency set above 1 while the M3 Ultra ran with concurrency 1 and prompt caching off, and the article states the M5 Ultra setting inconsistently (2 and 4).

generation:

49.1 tokens/s

quant:

4-bit (MLX)

User reports 49.1 t/s generation. Source evidence (verbatim): Source: https://arxiv.org/html/2605.00519v1 # First-hand M3 Ultra 96GB Qwen3-Next 80B native MLX 4-bit generation measurement Original experimental paper Silicon Showdown, arXiv 2605.00519v1, May 2026. Selected Table 5 row: Apple M3 Ultra 96GB, 28 CPU cores, 60 GPU cores; installed unified memory 96GB. macOS 26 Tahoe, MLX v0.30.6. Qwen3-Next-80B-A3B native MLX 4-bit: generation 49.1 tokens/s, 0% CPU offload. Long prompt approximately 496 tokens; output 256 tokens, greedy decoding. Authors report arithmetic mean of three consecutive timed runs after two untimed warmups and fixed random seed. This is their published generation-phase measurement, not a speed derived from TTFT. Prompt-processing speed unreported; leave promptTps null. Context capacity, KV quantization, speculation, concurrency and resident allocation unreported. Verified base-model reference explicitly cited by the paper: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct . This is not the tested MLX quantization repository; exact quantized artifact and revision unknown. Do not infer disk size or resident memory. Table 2 names the 96GB machine, not the followed 256GB configuration. Other models, machines and Nvidia offload comparisons are separate runs. Source: https://arxiv.org/html/2605.00519v1 .

generation:

29.0 tokens/s

prompt processing (prefill):

274.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.8-27B at 29 t/s generation and 274 t/s prompt processing on an M3 Ultra 96GB. Setup is llama.cpp 0.4.0 with the Q4_K_M GGUF, run natively through llama-bench on one Apple GPU via Metal. The native benchmark gives 28.97 +/- 0.03 tok/s generation and 273.98 +/- 0.03 tok/s prompt processing; context length, batch settings and offload flags are unreported.

generation:

109.7 tokens/s

prompt processing (prefill):

1770.1 tokens/s

quant:

4-bit (MLX)

User reports Qwen3.5-35B-A3B at 109.7 tokens/s generation and 1770.1 tokens/s prefill on an Apple Mac Studio M3 Ultra with 96 GB unified memory. Setup is MLX with 4-bit weights loaded via mlx_lm.load() and run through mlx_lm.generate.stream_generate() without an HTTP server, at a 512-token context setting, generating 50 tokens at temperature 0.0. Source reports 20.20 GB peak memory and 77.4 MB KV memory for this row. 512 is the tested context setting, not a configured maximum; runtime version, GPU core count, KV-cache precision and maximum context are not stated. This is a short-context standalone result, not an end-to-end server throughput measurement.

generation:

55-60 tokens/s

prompt processing (prefill):

667.0 tokens/s

quant:

Q4 (GGUF)

User reports Qwen3.8 Flash Next Q4 running through ds4 on an M3 Ultra 96GB with the full 262K context configured, delivering 55–60 tok/s decode and 667 tok/s prefill. About 80GB is used for model, KV cache and buffers, while the model's 95GB n-gram table streams from SSD. The user runs it daily through a codex harness as a headless box.