Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash MXFP4 at 36.1 t/s decode and 497 t/s prefill served across a Radeon 8060S iGPU and an RTX PRO 6000 Blackwell.
Setup is llama.cpp with the rocm-cuda backend on two GPUs, the Strix Halo iGPU plus an external RTX PRO 6000 96 GB, MXFP4 weights (146 GB) fully GPU-resident with no CPU offload and the experts of 18 layers on the iGPU, under Windows.
The repo also reports that on Blackwell a CUDA 13.3 build of llama.cpp loses 50-80% of decode past 8192 context: an unnamed 27B model on the RTX PRO 6000 drops from 72 to 35 t/s, while a CUDA 12.8 build of the same commit keeps 74.6 t/s at 8K. gpt-oss-120b runs 52 t/s on the iGPU alone, and Qwen3.8-Flash-Next Q4_K_M across two AMD GPUs gave 21.5-25.8 t/s decode at 4K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contexttool-use
User reports DeepSeek V4 Flash 162B (REAP K144 pruned) at 13.899 tok/s decode and 538.958 tok/s prefill on a single DGX Spark.
Setup is vLLM with NVFP4/MXFP4 expert weights and FP8 KV cache, 200K context, 14 GB KV cache, no speculative decoding.
Validation used a 186,390-token prompt with 345.834 s TTFT and needle retained; the watchdog logged a low-memory kill at teardown, so the profile is proof-of-concept. K160 prefills at about 514 tok/s and decodes at about 24 tok/s with MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-uselong-context
User reports DeepSeek-V4-Flash-0731 at ~9.8 tok/s decode on a single DGX Spark (GB10, 128GB unified memory).
Setup is a vLLM-based stack (vLLM-Moet adapted for unified memory and sm_121) with 2-bit expert planes and an FP4 tier, serving 256K context with a KV pool of ~660K tokens.
Decode uses the built-in DSpark speculative head at k=2; the recommended k=7 was slower at 5.1 tok/s. Prefill is ~500 tok/s chunked at 2048, with a 28K-token prompt giving ~53 s TTFT. First boot took ~35 min for one-time expert quantization, later boots ~10 min via persistent cache. Tool calling and per-request thinking toggle work out of the box.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visioncodingagenticlong-context
User reports DeepSeek-V4-Flash-Vision-Exp at 80.1 tok/s on count-to-100, 51.8 tok/s on code and 33.2 tok/s on prose, on 2x DGX Spark at TP2.
Setup is vLLM with NVFP4 weights and nvfp4_ds_mla KV cache, 1M context, DSpark speculative decoding at k=5, KV pool 2.79M tokens / 19.12 GiB at gpu_memory_utilization 0.85.
Without the Patch 4 shared-expert fix the same workloads measure 50.7, 47.2 and 30.4 tok/s. The 0731 text checkpoint reaches 78.4 tok/s peak at 98.9% acceptance with Patch 4, versus 66.1 peak and 60.2% acceptance without. The user notes a ~30% cold-start penalty and that realistic mixed agent traffic at concurrency 4 gives ~88.6 tok/s aggregate and ~22.3 tok/s per stream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagenticcoding
User reports DeepSeek-V4-Flash at ~41 tok/s single-stream and ~350 tok/s aggregate at c=32 on 2x DGX Spark (GB10, 128 GB unified) via vLLM TP=2 over QSFP56 RoCE.
Setup is vLLM with official FP8 weights, 1M context profile, shared ~1.1M-token KV pool; prefill ~1785 t/s.
Long-context sweep: 8k 40 t/s, 100k 40 t/s, 200k 38 t/s, 500k 32 t/s decode; prefill ~1970 down to ~1380 t/s. Cross-machine: RTX PRO 6000 ds4 46.9 t/s, M2 Ultra 29.7 t/s, single Spark ds4 IQ2_XXS ~14 t/s. llama.cpp GGUF suite up to 67 t/s (35B-A3B), 172B MoE 31 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports DeepSeek V4 Flash 0731 at 44-47 tok/s decode on one NVIDIA DGX Spark (GB10, 128 GiB unified memory).
Setup is vLLM with EXL3 3.0 bpw weights, NVFP4 KV cache, 384k context, DSpark K5 speculative decoding with a K64 draft model, single sequence.
The 44-47 tok/s is a range. A 370,104-token stress test achieved exact needle recall with 0 preemptions, prefill ~625 tok/s effective, 594 s end-to-end. KV pool is 439,622 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visioncodinglong-contextagentic
User reports a two-node DGX Spark recipe for DeepSeek V4 Flash Vision-Exp under vLLM with TP=2 and DSpark speculative decoding, giving ~62-83 decode tok/s on a single chat through 128K context.
Setup is vLLM with NVFP4 weights and nvfp4_ds_mla KV cache, 17.04 GiB KV pool holding 2,331,430 tokens at 0.835 GPU utilization, 1M-token ceiling and 6 concurrent sequences.
Six short chats reach ~160-190 tok/s aggregate (~30-37 per stream); a 2026-08-14 run measured 162 aggregate at c=6 and 75 tok/s at 128K single-stream, while a 2026-09-02 run measured 56 tok/s at 256 tokens and 139 aggregate at c=6. A three-Spark TP=3 lane gives ~200 tok/s aggregate at 16 streams with 4-13% faster decode per stream but 4-13% slower prefill to 64K and ~22% slower at 128K-256K. The 315/205 tok/s figures need the Stage-C plus Keys path.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-use
User reports DeepSeek-V4-Flash at ~28 tok/s single-stream decode on one RTX 5090 with dual Xeon Gold 6138 and 256GB DDR4-2400.
Setup is stock ktransformers + sglang with int4 AMXINT4 CPU-offloaded experts, NUMA-sharded, top-4 adaptive-k, 8192 context, fp8 KV cache. The 284B-A13B model is 142 GB at 4-bit and does not fit the GPU; experts are computed on the CPU.
Re-verified at 27.9-28.0 tok/s steady-state decode. Decode is DDR4-bandwidth-saturated. EAGLE speculative decode (27.2), more GPU experts (27.5), and 80 hyperthreads (2.7) all lost. Coding canary passes 8/8.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextcodingsummarizationmultilingual
User reports DeepSeek V4 Flash 294B Q8KXL at 7.2 t/s generation and 111.09 t/s prompt processing on a 10-GPU server (8x RTX 3090 + 2x RTX 5090, 256GB VRAM).
Setup is koboldcpp with llama.cpp under the hood, GGUF weights, all 10 cards allocated with tensor split, clocks locked to 1200 MHz and power limited to 200W per 3090 to prevent transient spikes.
The figures are from the first of seven large cybersecurity prompts (2769 input tokens, 813 output tokens); later prompts in the sweep drop to 4.28 t/s generation and 28.38 t/s prompt processing. The user notes the model was the smallest tested but punched above its weight in quality, and that the speed drop versus other models was an acceptable trade-off.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports DeepSeek V4 Flash 0731 running at 60-70 t/s on dual DGX Spark (ASUS GX10) with native quant.
Setup uses two DGX Spark units for agent work, including porting a ~170k LOC Node codebase to Rust.
The user asks what others use for coordinating multiple agents with local models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 4-bit served across two Mac Studio M3 Ultra (512GB) boxes: prefill 1012 tok/s on a 13.9K-token prompt (TTFT 13.7 s), decode 45-47 tok/s single-stream and 112-117 tok/s aggregate at batch 8.
Setup is MLX with an oMLX 0.6.3rc2 serving overlay, MTP/speculative machinery enabled, over a single Thunderbolt 5 cable using the jaccl RDMA mesh backend.
The decode figure is given only as a range, so no single generationTps is recorded. The repo is a reproducibility playbook; the oMLX 0.6.3rc2 build artifact is no longer available and must be obtained upstream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash 0731 at 24.48 t/s generation and 300.24 t/s prompt processing on 4x RTX 3090 (WRX80 platform, Threadripper PRO 3975WX, 251GB RAM).
Setup is llama.cpp (club-3090 build) with UD-IQ2_XXS GGUF quant, 204800 token context, MoE cache enabled, speculative decoding on (draft acceptance 0.60833, mean len 4.04), 4 GPUs with PCIe-only interconnect (no NVLink, P2P unreachable).
Model is 284.33B params, 84.62 GiB file at 2.56 BPW; GPUs are power-capped to 250W. Figures are from a single request (n_tokens 2797 prompt, 96 tokens generated).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at ~1.78 t/s generation and ~29.8 t/s prompt processing on dual RTX 3090s with 64GB DDR4 and 78GB swap.
Setup is llama.cpp with UD-Q8_K_XL GGUF weights, q8_0 KV cache, 65536-token context, 11 GPU layers and tensor split 1,1.
The user notes the generation figure used only GPU and RAM with no swap, and that prefill is fast because the GPUs handle it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks which inference engine is fastest for a full quant on 96GB of VRAM, with 4x RTX 3090 and 256GB of 8-channel DDR4.
They report getting at most 6 t/s on short prompts with llama.cpp CUDA.
No model, quant, or context length is named.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at about 150 tok/s prompt processing and 15 tok/s generation in short contexts on an RTX 3090 Ti 24GB with 96GB DDR5-6400 and an AMD 9950X CPU.
Setup is llama.cpp with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF, 98304 context, -ngl 99 and --n-cpu-moe 40, so 40 MoE layers ran on the CPU.
User says it works well in the pi harness with the llama.cpp chat template.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at 19.3 to 19.7 tokens/second single-stream on one NVIDIA DGX Spark (GB10, 128GB unified memory).
Setup is llama.cpp build 10235 with Unsloth UD-IQ2_M dynamic 2-bit weights, 131072 tokens of context per slot, flash attention on, and no speculative decoding.
Prefill runs 200 to 370 tok/s depending on request shape; four concurrent slots give 52 tok/s aggregate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 72.4 tok/s on two NVIDIA DGX Spark nodes.
Setup is vLLM with fp8_ds_mla KV cache, tensor parallel across both nodes over 200 GbE, prefix cache disabled, DSpark speculative drafting.
Also measured Qwen3.8-Flash-Next NVFP4 on SGLang with NEXTN speculation at 42.6 tok/s and GLM-5.3-Flash on SGLang with DFlash2 speculation at 41.1 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash 4bit at 29.9 tok/s generation and 426.5 tok/s prompt processing on a Mac Studio with M3 Ultra 512GB unified memory.
Setup is MLX 0.32.0 (draft PR #1189) with a standard 8,000-byte prompt, 300 generated tokens, temperature 0, and no speculative decoding.
The same machine also ran DeepSeek R1-0528 4bit MLX at 206.9 / 20.3, GLM-5.2 4bit MLX at 182.6 / 17.9, Kimi K2.7 Code Q2_K_XL GGUF on llama.cpp 571d0d5 at 144.8 / 24.6, Qwen3.5 397B-A17B 4bit MLX at 510.3 / 38.1, gpt-oss-120B MXFP4 MLX at 1,405.9 / 79.2, Qwen3.6 35B-A3B 4bit MLX at 2,892.2 / 94.6, Gemma 4 31B IT 4bit at 347.8 / 29.2, Gemma 4 26B-A4B IT 4bit at 2,508.5 / 104.9, Qwen3-Coder-Next 4bit at 2,099.8 / 77.2, and Devstral 2 123B 4bit at 90.4 / 8.9 (prompt / generation tok/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash at 106.1 tok/s single-request decode on 2x RTX Pro 6000 Blackwell (192GB VRAM).
Setup is vLLM with FP4+FP8 quantization and FP8 KV cache, no prompt caching and no speculative decoding, 1,024 output tokens per request. Single-request prefill at 8K context is 13,036 tok/s; decode falls to 96.3 tok/s at 1024K context.
System throughput with 8 concurrent requests is 367.8 tok/s at 1K context, 170.0 tok/s at 32K and 56.5 tok/s at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash on 2x RTX PRO 6000 Blackwell Max-Q 96GB (192GB total), with prompt processing 36.5-39.4 t/s and token generation 34.1-41.7 t/s.
Setup is llama.cpp wip/deepseek-v4-support branch with a native FP4 + FP8 GGUF (146GB, 4.39 bits per weight), ctx-size 8192, batch 4096, Flash Attention disabled, no speculative decoding, single request, short prompts of 14-50 tokens.
GPU utilization was 30-40% burst; power draw was 97.8W on GPU0 and 115W on GPU1.