M5 Ultra 256GB

generation:

80-90 tokens/s

prompt processing (prefill):

1600.0 tokens/s

quant:

oQ4e

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash at 80-90 t/s generation and 1.6k t/s prefill on a Mac Studio M5 Ultra 256GB (64-core), short of the 140-150 t/s and ~2k pp seen in oMLX benchmark recipes. Setup is oMLX 0.7.3rc1 (build 3517) with oQ4e quants, both Jundot's GLM-5.3-Flash-oQ4e-mtp and a self-quantized oQ4e from FP8. MTP was used, and DFlash2, patch-size changes and other tweaks were tried without reaching the target speeds.

generation:

200.0 tokens/s

prompt processing (prefill):

8000.0 tokens/s

quant:

Q6

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 200 tok/s decode and 8k tok/s prefill on a two-machine M5 Ultra 256GB (80-core) cluster. Setup is Tensorfold with Q6 weights, connected over two Thunderbolt 5 cables; the user copied Tensorfold's guide and had DeepSeek set it up. GLM-5.3 Flash on the same cluster gave 50 tok/s decode and 4.9k tok/s prefill, which the user found too slow. A previous Qwen3.8 27B Q4 run under NInfer gave around 1.4k prefill and 100-120 tok/s decode. The user runs Hermes as the harness and asks whether a lighter harness would speed things up.

generation:

61.7 tokens/s

prompt processing (prefill):

1587.5 tokens/s

quant:

mixed-4_8bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks GLM-5.3-Flash-MLX-mixed-4_8bit on an M5 Ultra 256GB with oMLX, reporting 61.7 t/s generation and 1587.5 t/s prompt processing at 1024-token prompt. Setup uses MLX mixed 4/8-bit weights with peak MLX memory of 174.17 GB; a 131k context prefill would need ~208.28 GB, exceeding the 205.80 GB prefill limit. At longer prompts generation stays 58.1-59.6 t/s and prefill 2323.9-2373.0 t/s; continuous batching reaches 67.3 t/s at 2x and 99.3 t/s at 4x.

generation:

141.0 tokens/s

prompt processing (prefill):

3906.4 tokens/s

quant:

Q6_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next at 141.0 t/s generation and 3906.4 t/s prompt processing on an M5 Ultra 256GB. Setup is oMLX with a Q6_K_XL GGUF at a 1024-token prompt, single request, no speculation. Decode ranges from 67.6 to 197.3 t/s across the 1024 to 200000-token prompt sweep. Continuous batching reaches 316.4 t/s aggregate at 4 concurrent requests. User says opencode runs fast and responsively but code quality still has room for improvement.

generation:

80-100 tokens/s

quant:

oQ4e

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM 5.3 Flash on an M5 Ultra 256GB with oMLX 0.7.1, gaining about 20-30 tok/s on long running tasks at single concurrency versus 0.7.0, which was around 60-80 tok/s. Setup is oMLX 0.7.1 with an oQ4e quant on a single machine. With Lightning MTP enabled the user gets anywhere from 80-100 tok/s.

generation:

25.5 tokens/s

prompt processing (prefill):

542.1 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash (10-17 checkpoint) running on a 256GB M5 Ultra (64-core GPU, 30-core CPU) at 20-25 tok/s generation and 500 tok/s prefill. Setup is oMLX with an oQ4e MLX build, 65% MoE expert offload, CED prefill on, SSD N-gram offload on and a 16384 prefill chunk size; the 4-bit N-gram table was swapped for the original FP8 shards. Single-request tables give 18.8-30.8 tok/s decode on an English novel and 21.2-25.9 tok/s on Python code, with peak memory up to 203.6 GB; context benchmarks show 532 tok/s prefill at 128k and 516 tok/s at 256k.

generation:

76.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM 5.3 Flash at 76 tok/s on a 256GB M5 Ultra Mac Studio, run locally with oMLX. Setup is oMLX on the Mac Studio; the user also runs Qwen 3.8 27B locally for other tasks. The user says the biggest weakness of local LLMs is being able to run only one instance of GLM, and that coding is a no-go locally, though images and video are fine. They also compare against Ollama Cloud's $20 plan and use Kimi K3 and DeepSeek 4.1 alongside GLM.

generation:

36.8 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash 4-bit with MTP at 36.8 tok/s decode on a disaggregated setup pairing an M5 Ultra 256GB (prefill) with an M3 Ultra 512GB (decode). Setup is MLX with the JACCL RDMA backend over two Thunderbolt 5 cables, mlx-vlm 0.7.6, a 17.5k-token prompt and 256 output tokens. Prefill runs on the M5 at 988 tok/s (17.8 s), the 358 MB cache is handed off over TB5 in 38 ms, and first token arrives on the M3 about 18.5 s after the request instead of about 40 s. Single-node figures: M3 Ultra 512GB prefill 438 tok/s at 2k chunks and 454 tok/s at 8k chunks, decode 32.9 tok/s; M5 Ultra 256GB prefill 1,046 tok/s at 2k chunks and 1,188 tok/s at 8k chunks, decode 47.2 tok/s. The user notes the M5 alone decodes faster for a single request, and that the prefill node needs the full model so this only works for models that fit in 256 GB.

generation:

80.3 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3-14B at 80.3 t/s on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is LM Studio with 4-bit MLX weights on one machine, 36 CPU cores and 80 GPU cores. The 80.3 t/s is the mean of three runs; TTFT is 3.53 seconds, and the displayed 2,754 prompt tokens/s is prompt-token count divided by TTFT, not native prefill. Configured context capacity is unknown.

generation:

13.9 tokens/s

prompt processing (prefill):

1237.0 tokens/s

quant:

BF16 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B at 13.9 t/s generation and 1237 t/s prompt processing on an M5 Ultra 256GB. Setup is MLX with BF16 weights at a 32768-token input and 128-token output, batch 1 on one device, with KV capacity unconfigured and blank. The 13.9 t/s is the median of five trials, with a range of 13.821 to 13.917 t/s; peak allocator memory was 71.978GB.

generation:

130.4 tokens/s

prompt processing (prefill):

7154.6 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 130.4 t/s generation and 7154.6 t/s prompt processing on an M5 Ultra 256GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output. The figures are separate five-trial medians, not pooled across lengths; the 32768-input run peaked at 22.079GB of allocator high-water memory.

generation:

74.7 tokens/s

prompt processing (prefill):

2710.5 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 74.7 t/s generation and 2710.5 t/s prompt processing on an M5 Ultra 256GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output. The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 72.259GB of allocator high-water memory.

generation:

67.0 tokens/s

prompt processing (prefill):

2529.8 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-14B-4bit at 67 t/s generation and 2529.8 t/s prompt processing on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is MLX via mlx-lm 0.31.3 with 4-bit MLX Safetensors, batch 1, no prompt cache, no draft model, and no KV or activation quantization. The figures are medians across five trials at a native 32768-token input and 128-token generation cell, with peak memory 14.332 GB; the 32768 context is the actual input length, not a supported maximum.

generation:

100.2 tokens/s

prompt processing (prefill):

2150.0 tokens/s

quant:

IQ4_NL (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 100.2 tok/s on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp b11538 with IQ4_NL weights, f16 KV cache, 262K context, and MTP speculative decoding with probabilistic drafting at n-max 5. Without speculation the model decodes at 57.0 tok/s; prompt processing reaches 2150 tok/s with ubatch 4096. The full 262K context fits with about 103 GB of GPU memory free. A second copy of the model loaded by another app caused swapping and dropped speed to 12 tok/s.

generation:

135.2 tokens/s

prompt processing (prefill):

2458.0 tokens/s

quant:

oQ6e (MLX)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next oQ6e at 135.2 t/s decode and 2,458 t/s prefill on an M5 Ultra 256GB, comparing oMLX 0.7.0 and 0.7.1.dev1. Setup is oMLX with MLX oQ6e weights, 8-bit KV cache, Lightning MTP on, 1K context, single request. Across 4K-200K contexts, average decode is 120.9 t/s and prefill 4,761 t/s; 2 parallel requests give 74.2 t/s per request and 4 parallel 41.0 t/s per request. GLM-5.3-Flash oQ4 reaches 97.9 t/s decode and 2,264 t/s prefill, and Flash-Next oQ4e 121.6 t/s decode and 5,450 t/s prefill.

generation:

61.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash (320B) at ~61 tok/s generation on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp with GLM's built-in MTP head (probabilistic drafting, 3 draft tokens); without MTP generation is 39.7 tok/s. The full 262K context fits with ~15 GiB to spare. Prompt reading is ~765 tok/s at 4K tokens and ~630 tok/s at 59K; a 29K-token prompt takes 45 s. Long-context recall was correct at 58.8K tokens. Images and MTP cannot be enabled simultaneously, and deep context on Metal needs an unmerged fix.

generation:

47.6 tokens/s

prompt processing (prefill):

1672.8 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 47.597 t/s generation and 1672.768 t/s prompt processing on an M5 Ultra 256GB Mac Studio. Setup is native mlx_lm.benchmark (mlx-lm 0.31.3 on MLX 0.32.2) with 4-bit MLX weights, 32768 prompt tokens and 128 generated tokens over 5 trials, peak memory 20.143 GB. Figures are medians from a third-party review unit; generation ranged 47.558-47.610 t/s and prompt processing 1671.556-1673.415 t/s. Speed only, no quality evaluation, and no hardware speedup ratio is claimed.

generation:

53.0 tokens/s

prompt processing (prefill):

2200.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 53 t/s generation and 2,200 t/s prefill on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is MTPLX 2.12.0 with speculative decoding and 55-65% draft acceptance; quantization and KV cache are unspecified. The 53 t/s figure is at roughly 4K context, with 51 t/s at roughly 16K context; the 2,200 t/s prefill is at a roughly 17K prompt. Resident memory was not measured; the 99 GB figure is the on-disk size only. Measurements are medians of three runs per cell, and the author notes small models served traffic on the same GPU throughout testing.

generation:

58.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 58 tok/s generation and 2,000 tok/s prefill on a Mac Studio M5 Ultra 256GB. Setup is MTPLX 2.12.0 with speculative decoding, 55 to 65 percent draft acceptance, 99 GB resident, 4k context. Four models run in slots on the same box; Gemma 4 26B 4-bit reaches 113 tok/s, Qwen 3.6 35B-A3B 114 tok/s, DeepSeek V4 Flash 4-bit 31 tok/s. DeepSeek prefill is from a single uncached 13,139-token request in 21.8 s. Clef-Flash 9B decision slot has a 259 ms median.

generation:

76.7 tokens/s

prompt processing (prefill):

2270.0 tokens/s

quant:

oQ4 (MLX)

kv:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks oMLX 0.7.0 against 0.7.0rc1 on an M5 Ultra 256GB, running GLM-5.3-Flash oQ4 at 4K–200K context. Prefill improves from 792 to 2,270 tok/s (median) and decode from 55.4 to 76.7 tok/s. A 1M-token prompt completes with prefill 1,485 tok/s, decode 41.5 tok/s, and time to first token 11.8 min. Setup uses Lightning MTP, TurboQuant KV 4-bit, and one model loaded at a time.