- reported speed:
- 41.1 tokens/s generation · 541.0 tokens/s prompt processing
- quant:
- Q2_K (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek V4.1 Flash at 41.1 tok/s decode on 8× NVIDIA A40 with layer-split execution.
Setup is the TensorSharp engine with Q2_K weights; prefill measured 532–541 tok/s with GPU-resident Engram, and cold model loading from MooseFS took 144–155 seconds.
Decode improved from 35.6 to 41.1 tok/s after reducing decode graph partitions from 565–577 to 8. These are project benchmarks, not a comparison against llama.cpp.
- reported speed:
- 40.7 tokens/s generation · 539.0 tokens/s prompt processing
- quant:
- Q2_K (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek V4.1 Flash at 40.3–40.7 t/s single-stream decode on 8× NVIDIA A40 GPUs with Q2_K GGUF weights.
Setup is TensorSharp with layer splitting, F16 KV cache, and 65K context; prefill is 533–539 t/s. Engram tables are ~60 GiB and kept on GPU.
With Q4_K_M, single-stream decode is 31.0–32.5 t/s and prefill 452–492 t/s; aggregate decode is 39.3 t/s at 2 concurrent, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beats routed-MoE tensor parallelism on this machine, ~32 t/s vs ~22 t/s, because the cards have no NVLink.
- reported speed:
- 32.5 tokens/s generation · 492.1 tokens/s prompt processing
- quant:
- Q4_K_M (GGUF)
- kv:
- F16
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek V4.1 Flash at 32.5 t/s single-stream decode on 8× NVIDIA A40 with Q4_K_M GGUF.
Setup is TensorSharp with layer split, F16 KV cache, and 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts.
Q2_K achieved 40.72 t/s decode and 539 t/s prefill. Q4_K_M aggregate decode throughput was 39.3 t/s at 2 concurrent requests, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beat routed-MoE tensor parallelism (31–32.5 vs. 21.4–22 t/s).
- reported speed:
- 31.5 tokens/s generation · 836.0 tokens/s prompt processing
- quant:
- Q8_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User benchmarks TensorSharp against llama.cpp on 4x A40 GPUs, with TensorSharp's CUDA backend reaching 836 t/s prefill at 16K context and 31.5 t/s decode short.
The model is DeepSeek-V4-Flash-0731-UD-Q8_K_XL from unsloth.