NVIDIA A40 48GB reports

generation:

22.9 tokens/s

prompt processing (prefill):

2567.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks MiniMax-M3.1-Flash 428B at 22.9 t/s decode and 2,567 tok/s prefill at 64K context on 8x NVIDIA A40 with TensorSharp. Setup is TensorSharp with Unsloth UD-Q4_K_XL GGUF (232.3 GiB of model files), F16 KV cache, layer splitting, and speculative decoding disabled. Against Unsloth's llama.cpp support branch on the same files, 64K prefill was 262 tok/s and decode 15.4 t/s; short-chat decode was nearly tied at 26.3 vs 26.0 t/s. Tensor parallelism reached about 41 t/s short-chat decode, and DSpark with confidence gating raised mean generation from 22.9 to 32.3 t/s.

generation:

22.9 tokens/s

prompt processing (prefill):

2567.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks MiniMax-M3.1-Flash at 22.9 tok/s decode at 64K context on 8x NVIDIA A40 GPUs. Setup is TensorSharp with the Unsloth UD-Q4_K_XL GGUF, F16 KV cache, layer splitting, and no speculative decoding. Prefill was 1,699 tok/s at 8K, 2,449 tok/s at 32K and 2,567 tok/s at 64K; time to first token on a 64K prompt was 40.0 s. Results are early, from one host and one quantization, and are compared against llama.cpp (llama-server with flash attention and --no-kv-unified).

generation:

35.5 tokens/s

prompt processing (prefill):

960.5 tokens/s

quant:

UD-IQ2_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 35.5 t/s decode after a ~1.8K-token prompt on 5x NVIDIA A40, with prefill at 960.5 t/s. Setup is TensorSharp with UD-IQ2_XXS GGUF weights, tensor parallel across five GPUs, thinking disabled and greedy decoding; the 125.9 GiB Engram table stayed in host memory, so the run used both GPU and system memory. Decode was 35-36 t/s without a drafter and 51-55 t/s with the DSpark Q8_0 drafter; the same machine's llama.cpp fork reached 29.6 t/s decode and 174.6 t/s prefill. Disabling the approximate YOCO prefill optimization gave 563.9 t/s prefill and perplexity 2.4811 versus 2.4978 with it on and 2.4697 for llama.cpp. A four-request batch took 21.2-25.5 s on TensorSharp and 43.3-46.8 s on llama.cpp.

prompt processing (prefill):

563.9 tokens/s

quant:

UD-IQ2_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks DeepSeek-V4.1-Flash-1017 on five NVIDIA A40s over PCIe, reporting 563.9 tok/s prefill at ~1.8K tokens with YOCO disabled. Setup is TensorSharp with UD-IQ2_XXS GGUF shards (~300.5 GB), tensor parallel 5, and the 125.9 GiB Engram table mapped in host memory under a 233.8 GiB host-memory limit. llama.cpp on the same files reached 174.6 tok/s at ~1.8K and 225.7 tok/s at ~7.3K, where TensorSharp gave 549.7 tok/s. Workload was real-text summarization, greedy decoding, 256 output tokens, three measured requests per prompt size.

generation:

41.1 tokens/s

prompt processing (prefill):

532-541 tokens/s

quant:

Q2_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 41.1 tok/s decode on 8× NVIDIA A40 with layer-split execution. Setup is the TensorSharp engine with Q2_K weights; prefill measured 532–541 tok/s with GPU-resident Engram, and cold model loading from MooseFS took 144–155 seconds. Decode improved from 35.6 to 41.1 tok/s after reducing decode graph partitions from 565–577 to 8. These are project benchmarks, not a comparison against llama.cpp.

generation:

40.3-40.7 tokens/s

prompt processing (prefill):

533-539 tokens/s

quant:

Q2_K (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 40.3–40.7 t/s single-stream decode on 8× NVIDIA A40 GPUs with Q2_K GGUF weights. Setup is TensorSharp with layer splitting, F16 KV cache, and 65K context; prefill is 533–539 t/s. Engram tables are ~60 GiB and kept on GPU. With Q4_K_M, single-stream decode is 31.0–32.5 t/s and prefill 452–492 t/s; aggregate decode is 39.3 t/s at 2 concurrent, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beats routed-MoE tensor parallelism on this machine, ~32 t/s vs ~22 t/s, because the cards have no NVLink.

generation:

32.5 tokens/s

prompt processing (prefill):

492.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 32.5 t/s single-stream decode on 8× NVIDIA A40 with Q4_K_M GGUF. Setup is TensorSharp with layer split, F16 KV cache, and 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts. Q2_K achieved 40.72 t/s decode and 539 t/s prefill. Q4_K_M aggregate decode throughput was 39.3 t/s at 2 concurrent requests, 48.9 t/s at 4, and 48.5 t/s at 8. Layer split beat routed-MoE tensor parallelism (31–32.5 vs. 21.4–22 t/s).

generation:

31.5 tokens/s

prompt processing (prefill):

836.0 tokens/s

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks TensorSharp against llama.cpp on 4x A40 GPUs, with TensorSharp's CUDA backend reaching 836 t/s prefill at 16K context and 31.5 t/s decode short. The model is DeepSeek-V4-Flash-0731-UD-Q8_K_XL from unsloth.