Local model performance on your hardware

generation:

41.1 tokens/s

prompt processing (prefill):

532-541 tokens/s

quant:

Q2_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 41.1 tok/s decode on 8× NVIDIA A40 with layer-split execution. Setup is the TensorSharp engine with Q2_K weights; prefill measured 532–541 tok/s with GPU-resident Engram, and cold model loading from MooseFS took 144–155 seconds. Decode improved from 35.6 to 41.1 tok/s after reducing decode graph partitions from 565–577 to 8. These are project benchmarks, not a comparison against llama.cpp.

generation:

32.5 tokens/s

prompt processing (prefill):

492.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 32.5 t/s single-stream decode on 8× NVIDIA A40 with Q4_K_M GGUF. Setup is TensorSharp with layer split, F16 KV cache, and 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts. Q2_K achieved 40.72 t/s decode and 539 t/s prefill. Q4_K_M aggregate decode throughput was 39.3 t/s at 2 concurrent requests, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beat routed-MoE tensor parallelism (31–32.5 vs. 21.4–22 t/s).

generation:

31.5 tokens/s

prompt processing (prefill):

836.0 tokens/s

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks TensorSharp against llama.cpp on 4x A40 GPUs, with TensorSharp's CUDA backend reaching 836 t/s prefill at 16K context and 31.5 t/s decode short. The model is DeepSeek-V4-Flash-0731-UD-Q8_K_XL from unsloth.