Local model performance on your hardware

generation:

22.9 tokens/s

prompt processing (prefill):

2567.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

User benchmarks MiniMax-M3.1-Flash 428B at 22.9 t/s decode and 2,567 tok/s prefill at 64K context on 8x NVIDIA A40 with TensorSharp. Setup is TensorSharp with Unsloth UD-Q4_K_XL GGUF (232.3 GiB of model files), F16 KV cache, layer splitting, and speculative decoding disabled. Against Unsloth's llama.cpp support branch on the same files, 64K prefill was 262 tok/s and decode 15.4 t/s; short-chat decode was nearly tied at 26.3 vs 26.0 t/s. Tensor parallelism reached about 41 t/s short-chat decode, and DSpark with confidence gating raised mean generation from 22.9 to 32.3 t/s.

generation:

22.9 tokens/s

prompt processing (prefill):

2567.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

User benchmarks MiniMax-M3.1-Flash at 22.9 tok/s decode at 64K context on 8x NVIDIA A40 GPUs. Setup is TensorSharp with the Unsloth UD-Q4_K_XL GGUF, F16 KV cache, layer splitting, and no speculative decoding. Prefill was 1,699 tok/s at 8K, 2,449 tok/s at 32K and 2,567 tok/s at 64K; time to first token on a 64K prompt was 40.0 s. Results are early, from one host and one quantization, and are compared against llama.cpp (llama-server with flash attention and --no-kv-unified).

generation:

35.5 tokens/s

prompt processing (prefill):

960.5 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports DeepSeek V4.1 Flash at 35.5 t/s decode after a ~1.8K-token prompt on 5x NVIDIA A40, with prefill at 960.5 t/s. Setup is TensorSharp with UD-IQ2_XXS GGUF weights, tensor parallel across five GPUs, thinking disabled and greedy decoding; the 125.9 GiB Engram table stayed in host memory, so the run used both GPU and system memory. Decode was 35-36 t/s without a drafter and 51-55 t/s with the DSpark Q8_0 drafter; the same machine's llama.cpp fork reached 29.6 t/s decode and 174.6 t/s prefill. Disabling the approximate YOCO prefill optimization gave 563.9 t/s prefill and perplexity 2.4811 versus 2.4978 with it on and 2.4697 for llama.cpp. A four-request batch took 21.2-25.5 s on TensorSharp and 43.3-46.8 s on llama.cpp.

prompt processing (prefill):

563.9 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User benchmarks DeepSeek-V4.1-Flash-1017 on five NVIDIA A40s over PCIe, reporting 563.9 tok/s prefill at ~1.8K tokens with YOCO disabled. Setup is TensorSharp with UD-IQ2_XXS GGUF shards (~300.5 GB), tensor parallel 5, and the 125.9 GiB Engram table mapped in host memory under a 233.8 GiB host-memory limit. llama.cpp on the same files reached 174.6 tok/s at ~1.8K and 225.7 tok/s at ~7.3K, where TensorSharp gave 549.7 tok/s. Workload was real-text summarization, greedy decoding, 256 output tokens, three measured requests per prompt size.

generation:

41.1 tokens/s

prompt processing (prefill):

532-541 tokens/s

quant:

Q2_K (GGUF)

User reports DeepSeek V4.1 Flash at 41.1 tok/s decode on 8× NVIDIA A40 with layer-split execution. Setup is the TensorSharp engine with Q2_K weights; prefill measured 532–541 tok/s with GPU-resident Engram, and cold model loading from MooseFS took 144–155 seconds. Decode improved from 35.6 to 41.1 tok/s after reducing decode graph partitions from 565–577 to 8. These are project benchmarks, not a comparison against llama.cpp.

generation:

40.3-40.7 tokens/s

prompt processing (prefill):

533-539 tokens/s

quant:

Q2_K (GGUF)

kv:

F16

User reports DeepSeek V4.1 Flash at 40.3–40.7 t/s single-stream decode on 8× NVIDIA A40 GPUs with Q2_K GGUF weights. Setup is TensorSharp with layer splitting, F16 KV cache, and 65K context; prefill is 533–539 t/s. Engram tables are ~60 GiB and kept on GPU. With Q4_K_M, single-stream decode is 31.0–32.5 t/s and prefill 452–492 t/s; aggregate decode is 39.3 t/s at 2 concurrent, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beats routed-MoE tensor parallelism on this machine, ~32 t/s vs ~22 t/s, because the cards have no NVLink.

generation:

32.5 tokens/s

prompt processing (prefill):

492.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports DeepSeek V4.1 Flash at 32.5 t/s single-stream decode on 8× NVIDIA A40 with Q4_K_M GGUF. Setup is TensorSharp with layer split, F16 KV cache, and 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts. Q2_K achieved 40.72 t/s decode and 539 t/s prefill. Q4_K_M aggregate decode throughput was 39.3 t/s at 2 concurrent requests, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beat routed-MoE tensor parallelism (31–32.5 vs. 21.4–22 t/s).

generation:

31.5 tokens/s

prompt processing (prefill):

836.0 tokens/s

quant:

Q8_K_XL (GGUF)

User benchmarks TensorSharp against llama.cpp on 4x A40 GPUs, with TensorSharp's CUDA backend reaching 836 t/s prefill at 16K context and 31.5 t/s decode short. The model is DeepSeek-V4-Flash-0731-UD-Q8_K_XL from unsloth.