Local model performance on your hardware

prompt processing (prefill):

2200.0 tokens/s

quant:

IQ2_XS (GGUF)

User reports prefill reaching 2,200 tokens/s on a local DeepSeek deployment (DeepSeek 1017 IQ2_XS) on an L40S 48GB with 1TB RAM. Setup is a Sol-hacked custom engine built on Strata and Maya, with IQ2_XS weights. User says cold prefill was fast but short incremental prefill passes between agent tool calls were slow in Codex; adding those to the optimization metrics cut end-to-end task time by around 15% with decode speed unchanged.

generation:

73.7 tokens/s

prompt processing (prefill):

1974.7 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4 Flash at 1974.7 t/s prefill and 73.7 t/s decode at 510K input on a single L40S 48GB, up from 1147.2 t/s prefill and 26.1 t/s decode with the official engine's auto settings. Setup is a modified Strata v0.2.6 build with IQ1_M weights, CPU experts kept on NUMA0 of dual EPYC 9754 with about 1TB system RAM, 16384-token prefill chunks, a 384MiB reconstructed-weight cache, and DSpark/MTP speculative decoding at a 0.15 confidence threshold. Decode peaks at 76.9 t/s at 32K input; the official engine with DSpark forced on reached 1278.4 t/s prefill and 22.9 t/s decode at 510K.