Local model performance on your hardware

generation:

30.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at ~30 tok/s decode at 2K context on an M5 Max 128GB, streaming a 341 GB Q2 GGUF from SSD. Setup is a fork of antirez's ds4 engine (sf-ds4-1flash) with Metal backend, one machine, weights streamed from disk. Decode is ~26 tok/s at 32K context and settles around 24 tok/s after an hour of heavy use. First token after an 8K prompt dropped from 1.23 s to 0.15 s. Prefill improved 45-99% and decode 62-78% versus upstream ds4. Output is bit-identical to ds4.

generation:

24.4 tokens/s

prompt processing (prefill):

636.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at Q2 running on an M5 Max 128GB Mac, with a fork of ds4 reaching 24.4 t/s generation at 2048 context versus 12.1 t/s for stock ds4. Setup is ds4 with a 341 GB GGUF (152 GB weights plus 189 GB Engram tables) streamed from SSD, no KV quant, bit-exact output. The fork doubles steady decode to 25.7 t/s at 2048 context and 22.0 t/s at 32768 context, and prefill from 16k to 32k context rises from 404 t/s to 636 t/s. Adding a byte-identical external Thunderbolt 5 SSD copy cuts time to first token on a 3.5K prompt from 13.2 s to 11.2 s and after an 8K context from 1.38 s to 0.18 s; decode is unchanged. The enclosure runs at PCIe 4.0 x4, about half the internal SSD speed. A two-SSD attempt in upstream ds4's mmap path was bit-exact but 13-38% slower on prefill and was not submitted.