DeepSeek V4.1 Flash 552B (16B active)
on 5× NVIDIA A40 48GB · TensorSharp · 1,800 ctx
Use cases
summarization
Summary
User reports DeepSeek V4.1 Flash at 35.5 t/s decode after a ~1.8K-token prompt on 5x NVIDIA A40, with prefill at 960.5 t/s.
Setup is TensorSharp with UD-IQ2_XXS GGUF weights, tensor parallel across five GPUs, thinking disabled and greedy decoding; the 125.9 GiB Engram table stayed in host memory, so the run used both GPU and system memory.
Decode was 35-36 t/s without a drafter and 51-55 t/s with the DSpark Q8_0 drafter; the same machine's llama.cpp fork reached 29.6 t/s decode and 174.6 t/s prefill. Disabling the approximate YOCO prefill optimization gave 563.9 t/s prefill and perplexity 2.4811 versus 2.4978 with it on and 2.4697 for llama.cpp. A four-request batch took 21.2-25.5 s on TensorSharp and 43.3-46.8 s on llama.cpp.