DeepSeek V4.1 Flash reports

generation:

60.8 tokens/s

prompt processing (prefill):

2574.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 60.8 t/s decode and 2,574 t/s prefill on a 1K prompt using a tuned SGLang build on 4x DGX Spark. Setup is SGLang TP4 with knapcio's recipe, DSpark speculative decoding, Engram tables streamed from NVMe, 262K context, and 100 GbE switch. The model weights are 511 GB and do not fit comfortably on 4 Sparks; memory allocation above 0.92 per Spark crashes. User compares against Patrick Moorhead's results: 62.5 t/s vs ~58 t/s single-request prose, 79.1 t/s vs ~96 t/s single-request code, and 293 t/s vs 494 t/s peak code at 16 vs 32 concurrent requests. User could not reproduce 1M context, 32 slots, or ~0.2 s first token.

generation:

103.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at up to 103 tok/s writing code on a cluster of two Lucebox workstations, each with a Gorgon Halo 495 128GB plus an R9700 AI PRO 32GB. The two boxes are joined by a ConnectX-5 Ex card in each with a single 100 GbE cable and no switch, talking over RDMA (RoCE v2) and running as one model, with every token passing through every layer on both boxes. Each step sends about 95 small messages between the boxes. The user says this is 1.9 times faster than a single Lucebox Zero 495 with 192 GB running the same engine, and that the first two-box run was only a quarter faster than one box. A single 128 GB box cannot keep all of the model's experts in memory, while two boxes have room for it.

generation:

30.0 tokens/s

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at ~30 tok/s decode at 2K context on an M5 Max 128GB, streaming a 341 GB Q2 GGUF from SSD. Setup is a fork of antirez's ds4 engine (sf-ds4-1flash) with Metal backend, one machine, weights streamed from disk. Decode is ~26 tok/s at 32K context and settles around 24 tok/s after an hour of heavy use. First token after an 8K prompt dropped from 1.23 s to 0.15 s. Prefill improved 45-99% and decode 62-78% versus upstream ds4. Output is bit-identical to ds4.

generation:

258-266 tokens/s

quant:

NVFP4 (NVFP4)

kv:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash M3 at 258-266 tok/s per user at C1 on a DGX Station GB300 with an RTX PRO 6000 Blackwell Max-Q expert sidecar. Setup is vLLM with NVFP4 Engram tables and NVFP4 KV cache, 265 hot experts on the GB300 and 3,960 cold experts on the sidecar, sharing the station with a video model and keeping 3.42M KV tokens. Aggregate throughput is 1,046 tok/s at C8 and 1,629 tok/s at C16, up from 583 and 821 without the sidecar; prefill is 44.5K tok/s (60K on real text). GSM8K-200 is 98.0% and needle passes at 114K.

generation:

19.3 tokens/s

prompt processing (prefill):

690-790 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 19.3 tok/s decode (C1) on a single RTX 3090 24GB with 503 GiB DDR4 and NVMe. Setup is vLLM 0.13 with EXL3 3.0 bpw quantization, fp8 KV cache, 262,144-token context, and CPU expert offload. All 15,360 routed experts are pinned in host RAM (190.5 GiB), with an AVX2 CPU tier computing cold decode misses and an elastic VRAM expert cache holding the hottest experts. Decode scales to 24.0 tok/s at 2 streams and 26.6 tok/s at 4 streams. Prefill ranges from 690 to 790 tok/s across 8k to 261k tokens. An experimental Intel Arc B70 expert tier reached 41.3 tok/s at 4 streams (+55%), but quality A/B is pending. Prefill fidelity failed the inherited guard (top-1 0.9856 vs 0.988 required, mean KL 0.0068 vs 0.00103 allowed).

generation:

26.5-27.2 tokens/s

prompt processing (prefill):

490-500 tokens/s

quant:

fp8 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a DeepSeek-V4.1-Flash fast path in oMLX reaching ~26.5–27.2 tok/s decode and ~490–500 tok/s prefill at ~2k context on an M3 Ultra. Setup is oMLX with MLX 0.32.2, fp8 weights, wired limit on, Engram stub backend; the stock language.py baseline was ~15.5 tok/s decode and ~337 tok/s prefill. Batched aggregate at B=8 is ~79.5 tok/s (~3x vs B=1); the mmap plus hot cache variant gives ~25.5 tok/s decode and ~481 tok/s prefill.

generation:

38.0 tokens/s

prompt processing (prefill):

902.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 37.95 tok/s single-stream decode across 8 categories and 902 tok/s cold prefill at 2K tokens on four DGX Sparks. Setup is vLLM TP4 across four DGX Sparks over ConnectX-7 RoCE, MXFP4 experts and MXFP8 dense weights, DSpark speculative decoding with k=5, the whole model in unified memory (56.6-68.9 GiB per Spark), 300K context window. Per category the single-stream decode was code 73.8 tok/s, JSON 52.1, math 50.9 and prose 24.4. Six streams reached 131.86 tok/s aggregate. Cold prefill was 1,539 tok/s at 32K and 1,194 at 64K.

generation:

87.7 tokens/s

quant:

MXFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 87.7 tok/s single-stream decode on four NVIDIA DGX Sparks (TP=4). Setup is SGLang with native MXFP4 experts and FP8 dense weights, DSpark speculative decoding (k=5), 1M context configured, 8M-token KV pool pinned, ~77 GiB resident weights per rank. Aggregate decode reaches 342.7 tok/s at 16 streams (22.2 per stream); cold prefill is 4,059-5,925 tok/s at 4k-262k context, 4.9k-5.0k tok/s on real text at 16k-128k. A 1,011,084-token needle test passed. The 3-Spark TP=3 profile is also documented at 37.9 tok/s single-stream.

generation:

24.4 tokens/s

prompt processing (prefill):

636.0 tokens/s

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at Q2 running on an M5 Max 128GB Mac, with a fork of ds4 reaching 24.4 t/s generation at 2048 context versus 12.1 t/s for stock ds4. Setup is ds4 with a 341 GB GGUF (152 GB weights plus 189 GB Engram tables) streamed from SSD, no KV quant, bit-exact output. The fork doubles steady decode to 25.7 t/s at 2048 context and 22.0 t/s at 32768 context, and prefill from 16k to 32k context rises from 404 t/s to 636 t/s. Adding a byte-identical external Thunderbolt 5 SSD copy cuts time to first token on a 3.5K prompt from 13.2 s to 11.2 s and after an 8K context from 1.38 s to 0.18 s; decode is unchanged. The enclosure runs at PCIe 4.0 x4, about half the internal SSD speed. A two-SSD attempt in upstream ds4's mmap path was bit-exact but 13-38% slower on prefill and was not submitted.

generation:

43.0 tokens/s

quant:

VQ-8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 43.0 tok/s speculative decode on a single NVIDIA DGX Spark (GB10, 128 GB unified memory), using a custom YoungAi C/CUDA engine. Setup is a 113.6 GB VQ-8 base GGUF plus a ~40 MB per-domain sidecar; the 14.1k-token Agent request ran greedy with speculative decoding at 3.04 tok/round. Pure decode on the same request is 28.9-29.4 tok/s, and short-prompt greedy is 30.5-30.7 tok/s. Prefill is 1,055 tok/s on a 12.5k prompt, 940 tok/s on the 14.1k Agent request, and 671 tok/s on a 106.7k prompt (159 s TTFT). Domain sidecars raise top-1 agreement vs the original by +2.8 to +3.7 points across finance, code, law, medicine and science; the post-training file flips decisions on the solving request but does not generalize to held-out days.

generation:

200.9 tokens/s

prompt processing (prefill):

1943.0 tokens/s

quant:

FP4 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 200.9 t/s decode and 1,943.0 t/s prefill on 4× RTX PRO 6000 Blackwell 96GB GPUs. Setup is SGLang with FP4 routed experts and FP8 components, DSpark block 5 speculative decoding, 524,288 context, 64 GiB DDR5 cache, NVMe offload, 8 slots, GPUs capped at 275 W over PCIe without NVLink. The 200.9 t/s is the C1 single-stream figure from a 45-case sweep; total decode across 8 concurrent streams reached 713.5 t/s. The 4M populated-KV gate passed with 4,063,744 tokens across eight 500k inputs. Long-context answer quality and a one-hour soak remain unqualified.

generation:

249.1 tokens/s

prompt processing (prefill):

17982.0 tokens/s

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 249.1 tok/s single-stream decode and 17,982 tok/s prefill on four RTX PRO 6000 Max-Q cards. Setup is vLLM with bfloat16 weights, fp8 KV cache, TP4/DCP1, RAM Engram tables, adaptive DSpark K7 speculation, 4096-token budget, 32 slots and a 131,072 context cap. C8 aggregate output is 805.5 tok/s and the C1 verifier rate is 98.58 steps/s; the previous Karmic Kraken release measured 230.6 tok/s, 748.3 tok/s, 17,503 tok/s and 89.87 steps/s respectively.

generation:

2.4 tokens/s

quant:

FP8 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash (552B backbone + 196B Engram, 384 experts top-6, FP8 + FP4) running on a single RTX 5060 8GB at ~1.6 tokens/s from disk only and ~2.4 tokens/s with a 16 GB RAM cache. Setup streams experts and Engram rows from NVMe using DeepSeek's reference code with a modified storage layer; only the dense part lives on the GPU (cap --vram_gb 7.3), with 240 experts (4.2 GiB) read per token. Batch size 1, text only, context limit 8192, prefill max 700 tokens. Live chat via the OpenAI-compatible server with a 16 GB RAM cache reaches 2.43 tokens/s; peak VRAM 7.06 GiB. A 311-token prompt takes 17.9 s. The user notes the speed-ups changed nothing in output (136/136 tokens identical) and that DeepSeek's untouched reference could not be run side by side because it cannot load the model in 8 GB.

generation:

104.9 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks DeepSeek V4.1 Flash at 104.86 tokens/s decode on six RTX PRO 6000 Blackwell 96 GB cards. Setup uses B12X linears and experts with PCIe IPC, prefill budget 1536, concurrency 1, no CPU offload. Decode holds 103.37 tokens/s at 32K and 103.84 tokens/s at 128K; TTFT is 2.383 s at 32K and 9.848 s at 128K. The exact 1M-token test (1,048,448 input, 128 forced output) completed in 170.08 s, down from 187.10 s. The final profile shortened 128K TTFT by 15.3%. Server decode rates were 101.5-103.1 tokens/s against client estimates of 103-105 tokens/s.

generation:

97.2 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 97.23 tok/s for a single interactive request on eight RTX 5090 32GB GPUs over PCIe with no NVLink. Setup is vLLM with MXFP4 quantization, 32,768-token context, TP8 plus expert parallelism, and 9-11 GiB/rank of decoder expert offload to host RAM (503 GiB installed, about 264-279 GiB shared). Aggregate output reaches 247.77 tok/s at eight concurrent requests and 368.55 tok/s at 128 concurrent text requests; 8K prefill at two concurrent requests is 3,145 tok/s. The checkpoint is about 476 GiB and its quantization is unchanged.

generation:

117.0 tokens/s

prompt processing (prefill):

6066.0 tokens/s

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 117 tok/s decode on one stream at 128k context and 6,066 tok/s prefill at 105k tokens on eight 64 GB CMP 170HX mining cards. Setup is vLLM with fp8 KV cache, PP=8, DSpark speculative decoding with 5 draft tokens, Engram tables in pinned host RAM, and 1M context. Decode drops to 96 tok/s at 512k. Eight concurrent streams reach 532 tok/s aggregate (66 per stream) at 105k. KV pool holds 6.17M tokens. One card was capped to 180 W after PCIe bus drops.

generation:

6-7 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 6-7 tok/s on a single RTX 3090 with 128GBx3 DDR4-2933 LRDIMM and a 4-bit quant. Setup uses ik_llama.cpp with experts streamed from SSD and on-demand expert streaming to GPU; there is a 100-200 token warmup while experts populate. User notes the machine is bottlenecked at 40GB/sec RAM throughput with three sticks and expects 20-30 tok/s with a fourth RAM chip. GLM 5.3 Flash support is also included but was slow.

generation:

73.4 tokens/s

prompt processing (prefill):

1010.0 tokens/s

quant:

Q3-G128-LSQ (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User asks whether oMLX benchmarks can be spoofed, citing a third-party oMLX benchmark entry for deepseek-v4.1-flash-q3-g128-lsq-mlx on an M5 Ultra with 256GB RAM, posted two days earlier, reporting about 73.4 tok/s generation and 1,010 tok/s prompt processing. The run is not the user's own; the figures come from a linked oMLX benchmark page and screenshot.

generation:

2.7 tokens/s

prompt processing (prefill):

337.0 tokens/s

quant:

FP4 (FP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 2.68 tok/s median decode (2.64-2.71 tok/s range) on a single NVIDIA DGX Spark (GB10) with 128 GB unified memory. Setup uses native FP4 experts, a 73.8 GB arena holding 25.6% of routed experts with the rest streamed from NVMe, DSpark speculative decoding, and MAX_SEQ 32768. Prefill is 337 tok/s on a 5,014-token prompt (14.9 s). Without DSpark decode is 1.75 tok/s. Expert hit rate is about 0.83 and about 0.9 GB is read from NVMe per token.

generation:

5.1 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User ports DeepSeek-V4.1-Flash (552B) to a llama.cpp fork and runs it on a single RTX 5090 with 31.8 GiB VRAM and 125.7 GiB RAM, streaming routed experts from NVMe through a VRAM cache and pinned host tier. Reports 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache. Setup uses MXFP4 experts as a lossless repack of the released blocks, with the engram tables (189 GiB of n-gram memory) kept on disk and read by the host. Logit correlation against the reference at 1401 tokens is 0.9967. Measured ceilings are 6.2 tokens/s with no disk miss and 21.3 tokens/s with everything resident. Per remap call at a 72 GiB host tier: 20% compute, 26% PCIe, 54% NVMe. A prefetch oracle gives +30% only at 5 tokens of look-ahead; block verification with a perfect draft gives +17-20%; an n-gram draft loses 33% on realistic content; the DSpark draft head shows 51-97% acceptance but is neutral on mixed content.