Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 43.6 tok/s generation on a short prompt and 262 tok/s prefill on a 20K-token prompt on an RTX PRO 6000 Blackwell Max-Q.
Setup is DwarfStar 4 (ds4.c) with an IQ2_XXS + Q2_K quant, single session, no MTP or speculative decoding.
Generation drops to 31.4 tok/s at 50K context; peak VRAM observed 93,142 MiB of 95,593 MiB (95%).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4-Flash running at ~4.5-5 tok/s decode on an M5 Pro with 48 GB unified memory.
Setup is a from-scratch MLX inference engine that keeps dense/attention weights resident at 8-bit and streams the 256 routed experts (mxfp4) from SSD into a device-side LRU cache, with CAP=1664 and MAX_CTX=32768.
Decode is byte-bound by SSD expert streaming, and prefill and decode both cost about 200 ms per token. Lossless speculative decoding is built but does not fit in 48 GB, projecting ~6-7.5 tok/s at 64 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at ~26 tok/s generation and ~190 tok/s prompt processing on a MacBook Pro M5 Max with 128 GB unified memory.
Setup is LM Studio with antirez's deepseek_v4 llama.cpp fork sideloaded, IQ2XXS GGUF weights, F16 KV cache, and the full 1048576 token context. The model is a 256-expert MoE with ~80 GB resident.
The figures are approximate and the run is a workaround: upstream llama.cpp does not yet support the deepseek_v4 architecture and LM Studio's MLX runtime rejects V4 GGUFs. Long prompts beyond 16k tokens are flushed in 16k chunks due to the n_batch cap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-uselong-context
User reports DeepSeek V4 Flash at ~22 tok/s decode and ~75 tok/s prefill on a Mac Studio M3 Ultra with 256GB unified memory.
Setup is vMLX with a mixed-precision MLX affine quantization (Q4 routed experts, Q8 attention and shared expert, fp16 norms) at group size 32, 173 GB on disk, 1,048,576 token context.
vMLX disables continuous batching for this model, so all requests go through SimpleEngine; the bundle requires a runtime patch to vMLX's jang_tools loader to avoid 8-bit expert wrapping and BOS-token loops.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 5.58 tok/s on 2x RTX 3090 with the offload MoE backend, versus 0.67 tok/s with the auto-selected hybrid backend.
Setup is FreeToken 0.1.2 with fp8 dense weights and fp4 experts, TP=2, moe-cache-auto resolving to 624 slots (5.7% residency). Decode measured over a streamed 60-token completion, first token excluded, after a warmup request.
The auto-selected hybrid backend is 8.3x slower than offload. Raising memory-ratio to 0.95 grew the expert pool to 779 slots (7.1% residency) for only +6.6% throughput (5.95 tok/s). Doubling CPU threads from 23 to 44 yielded only +15%. The user notes TP=2 is a regression for offloaded MoE per issue #62, so absolute numbers are pessimistic.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagentictool-use
User reports DeepSeek-V4-Flash-0731 at ~40–50 tok/s on long structured answers and ~31–35 tok/s on free-form prose and reasoning on two Radeon AI PRO R9700 32GB cards with 192 GB system RAM.
Setup is the affinity engine with a 2.875 bpw quant, 262K context fully allocated at load, and a DeepSeek-native speculative draft.
Prefill is ~1,260–1,360 tok/s at 4K–64K context. Needle-in-a-haystack passed 3/3 at 4K/16K/64K, and a 15-turn agentic task passed every check. The user notes RDNA4 (gfx1201) only, two cards max, and ~64 GB+ RAM needed for experts that don't fit on the cards. An experimental DeepSeek-V4.1-Flash branch runs at only ~14 tok/s on two cards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationlong-contextagentic
User reports DeepSeek V4-Flash-0731 at 33 tok/s single-stream on 2x RTX 3090 in a used quad-Xeon R940 server with 768 GB DDR4.
Setup is the Lvllmds4-x vLLM fork with the official MXFP4 safetensors checkpoint, fp8_ds_mla KV cache, 32K context, and DSpark speculative decoding with 5 draft tokens.
Aggregate throughput is 53-68 tok/s at 4 concurrent requests and 47-63 tok/s at 8, with per-user rates of 13-17 and 6-8 tok/s. Cold prefill plateaus at 420-480 tok/s, warm prefill reaches 2,030-10,200 tok/s, and the same box on ik_llama.cpp hybrid does 12.2 tok/s single-stream. The user notes the bottleneck is CPU-side DRAM bandwidth, not the GPUs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports DeepSeek V4 Flash REAP-150B at 2.8-3.4 tok/s on an RTX 3060 12GB with a Ryzen 9 7900 and 64GB DDR5-6000 under WSL2.
Setup is FreeToken with FP4 experts and an experimental disk tier called Overspill; the expert math runs on the CPU via the AVX-512 path and experts stream from disk through RAM. The model is about 85GB.
User compares against llama.cpp at 0.4-0.5 tok/s and Colibri at 1.1-1.2 tok/s, and reports TTFT of 102s for a 6.4k prompt, 43s for the first short prompt after cold start and 10s for the next short prompt. User notes the runs are not bit-identical because llama.cpp stores the non-expert weights in Q8_0 while FreeToken and Colibri use FP8, and asks others to test on different hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks whether tokens per second is the right community metric for LLM performance, suggesting task complexity per second instead.
They cite DeepSeek V4 Flash 0731 with DSPark on a Mac Studio under llama.cpp producing about 22-28 TPS, and note the model does a lot of reasoning.
The Mac Studio chip and memory are not stated, so no GPU is recorded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagenticlong-context
User reports running DeepSeek V4 Flash 0731 on a Mac Studio M3 Ultra with 512GB unified memory, served by oMLX with the MXFP4 MLX weights.
Setup uses oMLX with the Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX model and Hermes Agent pointed at the oMLX endpoint. An earlier Unsloth GGUF attempt ran at around 4-7 tok/s because it fell back to CPU inference.
The user hit a cache invalidation problem after 130K tokens, caused by the hot cache size being capped at 30GB, and reports tool-call failures in Unsloth Studio that were traced to the UI not sending the tool template.
User reports running DeepSeek V4 Flash 0731 with UD-Q4_K_XL quant on 2x RTX 3090 + 1x RTX 3060 (128GB RAM, ~60GB VRAM) using a custom llama.cpp fork. Generation speed is low to mid twenties t/s, prompt processing around 45 t/s with the moe-cache branch, and near 200 t/s with a separate IQ2_M prompt-processing model. Setup uses 167936 context, q8_0 KV cache, dflash speculative decoding, and MoE expert caching with mlock. User notes prompt processing remains a bottleneck and overall prompt handling only improves for prompts over 30K tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports DeepSeek V4 Flash Vision at up to ~95 tok/s on 4× CMP 170HX 64GB (256GB aggregate HBM2e).
Setup is vLLM with pipeline parallel 4, max_model_len 262144, FP8 KV cache, DSpark speculative decoding with 6 tokens, and max_num_seqs 2.
Normal coding workload runs ~50–70 tok/s, strong speculative-decoding periods ~80–90 tok/s, peak ~95 tok/s. Cards sit around 56–62GB VRAM each at 50–60°C. Next tests planned for Qwen3.8 Flash Next and GLM 5.3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 98.1 t/s single-stream decode on 4x CMP 170HX (64 GB HBM2e each), up from a 50.8 t/s baseline with DSpark speculative decoding.
Setup is vLLM with pipeline parallelism, native MXFP4+FP8 weights (~155.4 GiB), context verified to 1,047,736 tokens. At 100k context single-stream decode is 38.8 t/s with 14.6 s time to first token.
Aggregate throughput across 64 concurrent requests is 712.8 t/s with DSpark (472.0 t/s baseline), and 90.0 t/s at 100k context. Prefill across the 25k-77k context range is about 5,300 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares ROCm and Vulkan performance for DeepSeek V4 Flash 0731 and Qwen 3.8 Flash Next on a Radeon AI PRO R9700 32GB plus Strix Halo in llama.cpp.
DeepSeek V4 Flash 0731 with UD-IQ4_NL quant and a DSpark Q2K-Q4K drafter on ROCm reached 154.67 t/s prompt and 24.07 t/s generation on a 40,180-token needle prompt; Vulkan reached 131.97 t/s prompt and 20.44 t/s generation.
Qwen 3.8 Flash Next with UD-Q4_K-XL and no speculative decoding on ROCm reached 566.10 t/s prompt and 17.96 t/s generation; Vulkan reached 477.33 t/s prompt and 18.99 t/s generation. User concludes ROCm still wins for DeepSeek while Vulkan has closed the gap, and for Qwen ROCm leads on prompt processing but Vulkan leads on token generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 285B MoE at 60+ t/s on 10 RTX 3090s and 120+ t/s on 12 RTX 3090s.
Setup is FP4 experts with FP8 attention and 157GB of weights, using TP2xPP5 on 10 GPUs at a 240W cap and TP4xPP3 on 12 GPUs.
The 60+ t/s figure is on 10 GPUs. Context runs to 1M without offload and 4M with RAM offload, with long-context prefill around 3500 t/s when batched. Vision, speculative decoding and tool calls work. A Docker image and repo are provided.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash quantized to a custom 3-bit with llama.cpp PR #24162 running at 5-6 t/s.
The run is at an early stage, with stability and performance tradeoffs still being worked out.
User praises the model's intelligence, its efficiency as context length scales, and its low KV cache usage, and predicts it will dominate the 80-140 GB model space.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash on two ASUS GX10 (DGX Spark) units connected via RoCE with TP=2, decoding 49.4 t/s at 4K context, 43.0 t/s at 16K, 37.9 t/s at 32K, 42.5 t/s at 128K and 39.8 t/s at 256K, with prefill of 2050 t/s, 2150 t/s, 2130 t/s, 1920 t/s and 1680 t/s at the same context lengths.
Setup is a vLLM fork from local-inference-lab with MXFP8 x MXFP4 weights.
At concurrency 4 and 128K context the aggregate is about 40-42 t/s. The KV cache holds up to about 1M tokens of context, though the user typically runs 256K. The user reports roughly 280W at maximum load and says the model beats M2.7 and Stepfun 3.7 on a private benchmark, praising its high-context retrieval and reasoning. A Docker compose file is provided.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports DeepSeek V4 Flash at 193 t/s on Hopper.
Setup is vLLM with Canada-Quant quants and patched MTP code.
User also reports roughly 400 t/s with 4 concurrent threads.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at about 40 t/s single-stream and 350 t/s aggregate with 32 concurrent requests at 256k context on dual DGX Sparks.
The 350 t/s figure is an aggregate across the 32 concurrent requests.
User compares this to an RTX Pro 6000 at about 46 t/s and a Mac M2 Ultra 192GB at about 29 t/s, both at Q2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 11-13 t/s on an M3 Max 96GB with SSD streaming and iogpu.wired_limit_mb=86016.
Setup is antirez's ds4 engine with GGUF. TTFT is 3-5s after warmup.
Prefill of 36k tokens takes about 2.5 minutes.