Local model performance on your hardware

generation:

16.0 tokens/s

prompt processing (prefill):

300.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at 300 t/s prefill and 16 t/s decode on an M3 Ultra 512GB, running the Q2 quant on the DS4 engine. Tool calls were functional but the model made questionable choices, ignoring the code execution tool and searching the local drive for a remote file before finding the right path. The 18k-token Hermes prompt was insufficient to steer it. At around 110k tokens of context the decode rate only just began to dip below 16 t/s, which the user notes is a key advantage of DS4 over llama.cpp. The GPU ran at about 95% throughout. The user compares it unfavorably to GLM 5.3 Flash, which puts out roughly 21-19 t/s, and is hesitant to try Q4.

generation:

31.3 tokens/s

prompt processing (prefill):

813.0 tokens/s

quant:

Q4 (GGUF)

mtp (multi-token prediction):

on

User reports DeepSeek V4.1 Flash at 31.3 t/s decode at 8k context on an M3 Ultra 512GB, up from 16.6 t/s upstream. Setup is a forked ds4 engine with Q4 GGUF weights, DSpark multi-token speculative decoding, and a disk KV cache. At 300k context decode is 28.3 t/s, prefill on a 62k prompt is 813 t/s, and TTFT on a 23k system prompt is 31.1 s. With DSpark on, code generation reaches 40.5 t/s and agent answer phases 41.3 t/s. A 91-minute agent turn decoded 101k tokens with 4.6M tokens prefilled at 99.5% cache hit and 56 tool calls. Output is byte-identical to upstream under greedy decode.

prompt processing (prefill):

475.0 tokens/s

User reports DeepSeek V4 Flash serving on an M3 Ultra 512GB with the ds4 engine, with cold prefill improved from 392 to 475 t/s at 64k context through kernel optimizations. Cache prewarming with max_tokens:0 yields roughly 10x speedup for chat turns, from 6-20s down to 1.6s.