Local model performance on your hardware

generation:

100.2 tokens/s

prompt processing (prefill):

2150.0 tokens/s

quant:

IQ4_NL (GGUF)

kv:

f16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 100.2 tok/s on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp b11538 with IQ4_NL weights, f16 KV cache, 262K context, and MTP speculative decoding with probabilistic drafting at n-max 5. Without speculation the model decodes at 57.0 tok/s; prompt processing reaches 2150 tok/s with ubatch 4096. The full 262K context fits with about 103 GB of GPU memory free. A second copy of the model loaded by another app caused swapping and dropped speed to 12 tok/s.

generation:

61.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash (320B) at ~61 tok/s generation on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp with GLM's built-in MTP head (probabilistic drafting, 3 draft tokens); without MTP generation is 39.7 tok/s. The full 262K context fits with ~15 GiB to spare. Prompt reading is ~765 tok/s at 4K tokens and ~630 tok/s at 59K; a 29K-token prompt takes 45 s. Long-context recall was correct at 58.8K tokens. Images and MTP cannot be enabled simultaneously, and deep context on Metal needs an unmerged fix.

generation:

17.0 tokens/s

prompt processing (prefill):

1069.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next 8-bit on a Mac Studio M5 Ultra 256GB, reporting 17 t/s decode and 1,069 t/s prefill at 250k context on llama.cpp (Metal). Setup is llama.cpp with Q8_0 weights and MTP off; oMLX 0.7.0rc1 reaches 51 t/s decode and 1,365 t/s prefill at the same depth, and 88 t/s decode at 10k with MTP on. A cold 250k-token prompt took 234 s before the first token, while the cached next step took 0.64 s. DeepSeek-V4-Flash at 400k and 500k context took 934 s and 1,311 s cold. The user notes one overnight agent run per setup and that the DeepSeek engine comparison mixes two model variants.