Local model performance on your hardware

generation:

41.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek-V4.1-Flash-1017 at 41 tok/s on a single RTX 3090 with 192 GB of system RAM, averaged across multiple prompts. Setup is the user's delta-inference-engine (a Windows port of Xolotl, built on Strata) running nazeshinjite's Q2 GGUF natively on Windows, with the model streamed from a 366 GB download and a 1M context. The user says output matches DeepSeek's API next token 90.2% of the time on general text and 92.4% on code/math. The same table gives 33 tok/s with 128 GB RAM, and the needs list gives 31 tok/s at 16 GB VRAM and 17 tok/s at 12 GB VRAM. Other quants measured 38 to 42 tok/s.

generation:

62.2 tokens/s

User reports DeepSeek V4.1-Flash decoding at a median 62.2 t/s on four DGX Spark (GX10) units in the original IKEA Kallax setup, with prefill of 4,255 t/s at 8K and 4,268 t/s at 250K. Decode uses speculative decoding, which swings about ±10 t/s between runs, so decode was left out of the cooling comparison. Each run is a single 8-minute sweep from 1K to 250K tokens with the model already loaded. Across six cooling setups the hottest GPU peaked at 82°C with no throttling; clocks stepped down about 3% from roughly 2525 MHz at 53°C to 2450 MHz at 82°C, and 250K prefill stayed between 4,225 and 4,353 t/s. The user's V3 open-faced air duct enclosure was the coolest setup at 65.1°C average under load.

generation:

23.3 tokens/s

prompt processing (prefill):

1077.0 tokens/s

quant:

IQ1_M (GGUF)

User reports DeepSeek V4.1 Flash at 23.3 t/s decode and 1,077 t/s prefill on a Strix Halo 128GB (Bosgame M5), fully resident in GPU memory with no SSD streaming. Setup is the user's Gufo fork with IQ1_M GGUF weights, a 112 GiB backbone in GPU memory, IQ4_NL Engram tables (103 GiB) read from NVMe, and a 4.6 GiB Q2_K/Q4_K drafter. The reported figure is plain decode; DSpark speculative decoding with prompt lookup reaches 28.0 t/s on mixed text and 65.3 t/s on repetitive text at depth 0. At 32K depth decode is 22.9 t/s with 993 t/s prefill, and at 128K 21.7 t/s with 700 t/s prefill. Eight concurrent users total 78.5 t/s, and editing a source file through prompt lookup gives 67.6 t/s. The user warns the setup needs the whole machine, leaving about 1.75 GiB of RAM with the drafter loaded.

generation:

106.0 tokens/s

quant:

EXL3 3.5bpw (EXL3)

User reports DeepSeek V4.1 Flash at 106 t/s per stream at concurrency 1 on four DGX Sparks, averaging 93 t/s aggregate across eight categories. Setup is TensorFold's Zig engine with an EXL3 3.5bpw pack, 1M context, a 2M-position KV pool, DSpark speculative drafts on and tensor parallel across all four GB10 nodes over RoCE v2. The sweep runs to 16 streams: per-stream decode falls to 20 t/s while aggregate rises to 250 t/s at C16. Coding reaches 139 t/s per stream at C1 and 343 t/s aggregate at C16; prose 81 and 224. Cold prefill was skipped, previously measured at 1.9-2.3K tok/s.

generation:

35.0 tokens/s

quant:

IQ2_XXS (GGUF)

User asks whether 2x AMD R9700 with 128GB RAM running GLM-5.3-Flash or DeepSeek-V4.1-Flash under Strata, Maya or Xolotl would beat a single RTX 5090 with 128GB RAM. The only figure given is the 5090 setup at roughly 35 tok/s with Strata on deepseek-v4.1-flash-1017-gsq-rco IQ2_XXS. No benchmark of the dual R9700 configuration is reported.

generation:

10.0 tokens/s

User reports DeepSeek V4.1 Flash running at about 10 t/s at 8K context on an RTX 5090. Setup is Strata with a Lexar 4TB NVMe and 64 GB of DDR5 system RAM. User says the speed makes the API cheaper than the electricity cost, but is glad it runs at all on the PC.

generation:

35.5 tokens/s

prompt processing (prefill):

960.5 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports DeepSeek V4.1 Flash at 35.5 t/s decode after a ~1.8K-token prompt on 5x NVIDIA A40, with prefill at 960.5 t/s. Setup is TensorSharp with UD-IQ2_XXS GGUF weights, tensor parallel across five GPUs, thinking disabled and greedy decoding; the 125.9 GiB Engram table stayed in host memory, so the run used both GPU and system memory. Decode was 35-36 t/s without a drafter and 51-55 t/s with the DSpark Q8_0 drafter; the same machine's llama.cpp fork reached 29.6 t/s decode and 174.6 t/s prefill. Disabling the approximate YOCO prefill optimization gave 563.9 t/s prefill and perplexity 2.4811 versus 2.4978 with it on and 2.4697 for llama.cpp. A four-request batch took 21.2-25.5 s on TensorSharp and 43.3-46.8 s on llama.cpp.

generation:

7.4 tokens/s

quant:

FP4

User reports the full DeepSeek-V4.1-Flash-1017 model at up to 7.4 tok/s warm (5.8 cold) on a 160-token code prompt, on an RTX 3060 12GB with 64 GB DDR4 and two NVMe drives. Setup is the user's own PROMETHEVS engine, which streams the 510 GB original FP4 weights through VRAM, RAM and both drives, computing tokens on a 2-bit copy of the experts and recomputing uncertain tokens with the original weights; DeepSeek's draft head proposes blocks of five tokens. Server runs give 5-8 tok/s on code and 4.5-5.4 on reasoning and agent prompts, against 2.14 tok/s for plain exact decoding of the full model on the same PC. Agreement with the full weights is 97.4% for the 2-bit copy alone and 98.3% with corrections over 960 audited tokens. Only tested on this one Windows machine so far, needs about 1.1 TB of NVMe, and has not been compared with DeepSeek's reference implementation.

prompt processing (prefill):

563.9 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User benchmarks DeepSeek-V4.1-Flash-1017 on five NVIDIA A40s over PCIe, reporting 563.9 tok/s prefill at ~1.8K tokens with YOCO disabled. Setup is TensorSharp with UD-IQ2_XXS GGUF shards (~300.5 GB), tensor parallel 5, and the 125.9 GiB Engram table mapped in host memory under a 233.8 GiB host-memory limit. llama.cpp on the same files reached 174.6 tok/s at ~1.8K and 225.7 tok/s at ~7.3K, where TensorSharp gave 549.7 tok/s. Workload was real-text summarization, greedy decoding, 256 output tokens, three measured requests per prompt size.

prompt processing (prefill):

400.0 tokens/s

quant:

EXL3 2.9bpw (EXL3)

User reports DeepSeek V4.1 Flash 1017 running at up to 40 tok/s, averaging around 30 tok/s, on 2x CMP 170HX plus an RTX 5060 Ti 16GB (144GB total VRAM) with 128GB DDR4-3000 and an Intel Core i9-10980XE. Setup uses LibertAIDAI's EXL3 2.9 bpw quantization with DSpARK enabled and expert caching: 93% of routed experts run on the GPUs, the rest on the CPU, with frequently used experts migrated to VRAM in the background. Prefill is around 400 tokens/sec, which the user calls slow. The 40 tok/s is a peak; the average is about 30 tok/s.

generation:

7.4 tokens/s

quant:

2-bit

User reports DeepSeek-V4.1-Flash-1017 at 7.4 tok/s with a warm cache and 5.8 tok/s cold on an RTX 3060 12GB with 64 GB DDR4 and NVMe storage. Setup is a custom engine that computes tokens on a 2-bit copy of the experts, recomputes flagged tokens with the original FP4 weights, and uses a draft head proposing blocks of five tokens. The 510 GB model streams from disk; plain exact decoding on the same machine gives 2.14 tok/s. Reasoning and agent prompts run at 4.4 to 5.2 tok/s. On 960 audited tokens the 2-bit copy matches the exact path 97.4% of the time, 98.3% with corrections; a 1,101-token agent prompt scores 95.9%. The engine is not yet released.

generation:

25.5 tokens/s

prompt processing (prefill):

542.1 tokens/s

quant:

oQ4e (MLX)

User reports DeepSeek V4.1 Flash (10-17 checkpoint) running on a 256GB M5 Ultra (64-core GPU, 30-core CPU) at 20-25 tok/s generation and 500 tok/s prefill. Setup is oMLX with an oQ4e MLX build, 65% MoE expert offload, CED prefill on, SSD N-gram offload on and a 16384 prefill chunk size; the 4-bit N-gram table was swapped for the original FP8 shards. Single-request tables give 18.8-30.8 tok/s decode on an English novel and 21.2-25.9 tok/s on Python code, with peak memory up to 203.6 GB; context benchmarks show 532 tok/s prefill at 128k and 516 tok/s at 256k.

generation:

39.3 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports DeepSeek-V4.1-Flash at 39.3 tok/s on code and 24.6 tok/s on prose with 4x RTX 3090 (96 GB VRAM) plus an EPYC 9575F and 768 GB DDR5. Setup is llama.cpp built from PR #30817 with the UD-Q4_K_XL GGUF quant, 65536 context, flash attention on, and the DSpark Q8_0 drafter (8 GB) for speculative decoding with up to 3 draft tokens. All figures are one request at a time with thinking off, single runs. Prefill and generation on an 8K prompt were 326 / 20.6 tok/s and on a 32K prompt 363 / 25.4 tok/s. DSpark roughly doubles code speed with about 80% of guesses accepted, but only about 30% are accepted on prose, where it is sometimes a bit slower; the user keeps it on because they mostly use the model for code.

generation:

26.0 tokens/s

prompt processing (prefill):

170.0 tokens/s

quant:

2.6bit

User reports DeepSeek V4.1 Flash at 26 t/s generation and 170 t/s prompt processing on 2x RTX 3090s. Setup is the dsx engine with a 2.6-bit quant (2.6127 bpw experts) and Engram tables, at 262,144 context; the 187.6 GB weights plus 203 GB Engram tables ran from a SATA SSD, and loading from an HDD took 90-110 minutes. The build scores KL 0.1972, 82.59% top-1 and perplexity 9.3626 against the FP8/FP4 teacher; the engine does not support images yet.

generation:

27.0 tokens/s

prompt processing (prefill):

180-320 tokens/s

quant:

GSQ Uncensored

User reports DeepSeek-V4.1-Flash (552B) at 27 tok/s decode on a single RTX 4090, with prefill of 180-320 tok/s. Setup is the Xolotl engine (based on Strata) with a 152 GiB uncensored GSQ quant, 512K context, weights spread across 24GB VRAM, 90GB DDR4 and NVMe on a Ryzen 7 5700X. User also reports 21/21 HumanEval, 10/10 tool calls and 3/3 agentic bug-fix loops through the real server with thinking on, and says the engine also runs GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B.

generation:

56.8 tokens/s

quant:

EXL3 2.9bpw (EXL3)

User reports DeepSeek-V4.1-Flash-1017 at 56.8 tok/s decode on text and 92.2 tok/s on code with TensorFold on two DGX Spark, TP=2 over the CX7 cable. Setup is TensorFold with EXL3 2.9bpw weights and 1M context, using a speculative two-node decoder; the same pair with the vLLM kit reached 27.0 tok/s text and 39.9 tok/s code. Four concurrent code requests total 140 tok/s, and a 64K-token needle test took 35.9 s versus 80.4 s on the kit. Long-prompt prefill is 30-37% below the published numbers (1,707 vs 2,421 tok/s at 32K), and the first long prompt after a start costs about 6 s extra.

generation:

60.8 tokens/s

prompt processing (prefill):

2574.0 tokens/s

User reports DeepSeek V4.1 Flash at 60.8 t/s decode and 2,574 t/s prefill on a 1K prompt using a tuned SGLang build on 4x DGX Spark. Setup is SGLang TP4 with knapcio's recipe, DSpark speculative decoding, Engram tables streamed from NVMe, 262K context, and 100 GbE switch. The model weights are 511 GB and do not fit comfortably on 4 Sparks; memory allocation above 0.92 per Spark crashes. User compares against Patrick Moorhead's results: 62.5 t/s vs ~58 t/s single-request prose, 79.1 t/s vs ~96 t/s single-request code, and 293 t/s vs 494 t/s peak code at 16 vs 32 concurrent requests. User could not reproduce 1M context, 32 slots, or ~0.2 s first token.

generation:

103.0 tokens/s

User reports DeepSeek V4.1 Flash at up to 103 tok/s writing code on a cluster of two Lucebox workstations, each with a Gorgon Halo 495 128GB plus an R9700 AI PRO 32GB. The two boxes are joined by a ConnectX-5 Ex card in each with a single 100 GbE cable and no switch, talking over RDMA (RoCE v2) and running as one model, with every token passing through every layer on both boxes. Each step sends about 95 small messages between the boxes. The user says this is 1.9 times faster than a single Lucebox Zero 495 with 192 GB running the same engine, and that the first two-box run was only a quarter faster than one box. A single 128 GB box cannot keep all of the model's experts in memory, while two boxes have room for it.

generation:

30.0 tokens/s

quant:

Q2 (GGUF)

User reports DeepSeek V4.1 Flash at ~30 tok/s decode at 2K context on an M5 Max 128GB, streaming a 341 GB Q2 GGUF from SSD. Setup is a fork of antirez's ds4 engine (sf-ds4-1flash) with Metal backend, one machine, weights streamed from disk. Decode is ~26 tok/s at 32K context and settles around 24 tok/s after an hour of heavy use. First token after an 8K prompt dropped from 1.23 s to 0.15 s. Prefill improved 45-99% and decode 62-78% versus upstream ds4. Output is bit-identical to ds4.

generation:

19.3 tokens/s

prompt processing (prefill):

690-790 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

kv:

fp8

User reports DeepSeek-V4.1-Flash at 19.3 tok/s decode (C1) on a single RTX 3090 24GB with 503 GiB DDR4 and NVMe. Setup is vLLM 0.13 with EXL3 3.0 bpw quantization, fp8 KV cache, 262,144-token context, and CPU expert offload. All 15,360 routed experts are pinned in host RAM (190.5 GiB), with an AVX2 CPU tier computing cold decode misses and an elastic VRAM expert cache holding the hottest experts. Decode scales to 24.0 tok/s at 2 streams and 26.6 tok/s at 4 streams. Prefill ranges from 690 to 790 tok/s across 8k to 261k tokens. An experimental Intel Arc B70 expert tier reached 41.3 tok/s at 4 streams (+55%), but quality A/B is pending. Prefill fidelity failed the inherited guard (top-1 0.9856 vs 0.988 required, mean KL 0.0068 vs 0.00103 allowed).