Local model performance on your hardware

generation:

79.0 tokens/s

quant:

2.875 bpw

User reports DeepSeek-V4-Flash-0731 at 79 tok/s greedy decode on four Radeon AI PRO R9700 cards. Setup is the affinity engine with a prebuilt 2.875 bpw quant, all 11,008 experts resident in VRAM with no host pool allocated, on a Threadripper PRO 9955WX with 128 GB RAM. The same box on two cards gave 36 tok/s with 45% of experts in VRAM and the rest streaming from a 51 GB pinned host pool. Sampled decode at temp 1.0 was 63 tok/s and about 90 tok/s on a chat answer with thinking on. Load takes about a minute, 42 seconds of it pushing 96 GB of experts onto the cards.

generation:

19.9 tokens/s

prompt processing (prefill):

270.1 tokens/s

quant:

Q4_K experts / mixed F16-Q8 (GGUF)

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash at 19.9 t/s generation and 270.1 t/s prompt processing on two AMD Strix Halo nodes with 128 GB RAM each, linked by OdinLink USB4/TB5 RDMA. Setup is the Antirez Q4_K experts build with F16 HC/compressor/indexer and Q8 attention/shared/output, tensor parallel across two Radeon 8060S iGPUs with a balanced 50/50 expert split. The run used a fixed 2,048-token prompt followed by 300 generated tokens, with ordinary decode and no speculative decoding; the source gives 19.89 t/s and 270.10 t/s exactly.

generation:

18.6 tokens/s

prompt processing (prefill):

140.3 tokens/s

quant:

IQ3_XXS (GGUF)

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash IQ3_XXS at 18.6 t/s generation and 140.3 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 434ddbb with Vulkan on a single Radeon 8060S, 120GB GTT region, and a 97.05GiB tensor size. The figures are separate PP512 and TG128 synthetic tests, not a 512-input/128-output client request; the source gives 18.56 +/- 0.01 t/s and 140.30 +/- 4.35 t/s over 3 repeats.

prompt processing (prefill):

454.6 tokens/s

quant:

IQ2_XXS + Q2_K, selected Q8 tensors (GGUF)

User reports DeepSeek V4 Flash prefill at 454.6 t/s on an AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151) with 128 GB unified memory. Setup is the DS4 Halo ROCm prefill fork with IQ2_XXS + Q2_K and selected Q8 tensors, 4,352-token context allocation and 4,096-token prefill chunks. The figure is a warmed prepared-prefill mean of 454.59 t/s from two samples, 454.52 and 454.65, on a complete position-zero 4,096-token request with generation disabled.

generation:

17.8 tokens/s

prompt processing (prefill):

238.0 tokens/s

quant:

UD-IQ3_S

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash 284B-A13B at 17.8 t/s generation and 238 t/s prompt processing on an AMD Gorgon Halo 192GB machine. Setup is llama.cpp with UD-IQ3_S, Vulkan backend, all layers on GPU, on a Minisforum MS-S1 MAX-P495 with Ryzen AI Max+ PRO 495 and 192 GB unified memory. The 238 t/s and 17.8 t/s come from separate llama-bench pp512 and tg128 tests, not a paired 512-input/128-output conversation; the same source reports roughly 32k-depth prompt processing at 106 t/s and generation at 13.2 t/s.

generation:

15.4 tokens/s

prompt processing (prefill):

142.3 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

User reports DeepSeek-V4-Flash at 15.4 tok/s decode and 142.30 tok/s prefill at 2,048 tokens of context on a Ryzen AI Max+ PRO 395 (Strix Halo) with 128 GB unified memory. Setup is the ds4 engine with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF (81 GB), 32,768-token context, ROCm 7.1.1; decode falls 12% from 2K to 32K while prefill stays flat. A second model, Qwen3.8-Flash-Next UD-IQ4_XS (88 GB) under Strata 0.1.40, decodes at 39.0 to 40.9 tok/s and prefills a 6,786-token prompt at 343 to 367 tok/s, dropping to 30.4 tok/s each with two requests in parallel. The same engine measured 8.7 tok/s while another model's worker held 41 GiB of VRAM and 23 GB of RAM, against 39.0 tok/s on an exclusive box.

generation:

31.7 tokens/s

quant:

IQ2_XXS/Q2_K (GGUF)

User reports DeepSeek V4 Flash Vision-Exp at 31.7 tok/s decode on a Mac Studio M4 Max 128GB. Setup is ds4 (DwarfStar) with IQ2_XXS/Q2_K GGUF, DSpark speculative decoding at confidence 0.6, and a 32768 context length. Thinking-on pagoda run reached 29.84 tok/s over 26,415 completion tokens; 1M context loaded at 97.15 GiB planned total with 32.0 tok/s decode. User notes DSpark adds roughly 10-20% over no speculation.

generation:

36.1 tokens/s

prompt processing (prefill):

498.3 tokens/s

quant:

2.44bit-mixed (MLX)

User reports DeepSeek V4 Flash 0731 at 36.1 t/s generation and 498.3 t/s prefill on a MacBook Pro M5 Max 128GB 40 GPU. Setup is oMLX 0.5.7 with a 2.44 bpw mixed MLX quantization (92.8 GB on disk) at 1k prompt, single request, 128 generated tokens, speculative decoding off. Generation holds 33.8 t/s at 4k and 31.5 t/s at 32k, with peak memory rising 1.03 GB across an 8x context jump; prefill at 32k takes 97 seconds to first token. Continuous batching reaches 83.2 t/s at batch 8 (2.30x).

generation:

45.9 tokens/s

prompt processing (prefill):

362.3 tokens/s

quant:

q2-imatrix (GGUF)

User reports DeepSeek-V4-Flash q2-imatrix at 45.9 t/s generation and 362.3 t/s prefill on an RTX Pro 6000 Blackwell 96GB at 2048 context. Setup is ds4 with q2-imatrix GGUF, fp16 dequant cache disabled and MTP off, fully resident at ~91 GB peak. The 80.76 GB model only fits with DS4_CUDA_Q8_F16_CACHE_MB=0; MTP adds 3.5 GB and pushes over 96 GB. Generation beats posted M5 Max and DGX Spark GB10 q2 numbers; prefill is about equal to Spark. At 28672 context generation drops to 40.6 t/s. The user notes the card is tight for this model and heavier ds4_test sessions OOM.

generation:

28.9 tokens/s

prompt processing (prefill):

261.7 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports DeepSeek-V4-Flash-0731 (284B MoE, 13B active) at 28.91 t/s generation and 261.7 t/s prompt processing on a single AMD Ryzen AI MAX+ 395 with 128GB unified memory. Setup is a llama.cpp fork (Nathanw1014/llama.cpp) with Vulkan/RADV backend, UD-IQ2_XXS quant (2.06 bpw), DSpark draft model at n=3, 16384 context, 120 GiB GTT limit. Best result is IQ2_XXS + DSpark n=3 at 28.91 t/s, reaching 90% of Lucebox's 32.0 t/s with 6 experts instead of 4. The fork beats upstream by 49% on prompt processing at 8192 context. IQ2 with draft model sits at the memory limit and is a measurement setup, not a production state.

generation:

16.3 tokens/s

prompt processing (prefill):

484.6 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

User reports DeepSeek V4 Flash at 484.62 t/s prefill and 16.28 t/s decode on a Strix Halo 128GB (Ryzen AI MAX+ 395, Radeon 8060S, gfx1151). Setup is Gufo with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 mixed GGUF, one session, speculative decoding disabled, thinking off, temperature 0, seed 1, nominal pp2048/tg128 (2,035 prompt tokens, 128 generated). One warmed sample; no standard deviation reported. The maintainer notes target-model quality parity is unestablished and build-sensitive differences are under investigation. Power, KV-cache type, Flash Attention and peak memory were not reported.

generation:

40.0 tokens/s

User reports DeepSeek V4 Flash reaching a peak of 40 t/s generation on 8x Radeon AI PRO R9700 under llama.cpp, but the run froze after 200-300 tokens. Setup is llama.cpp on 8x R9700 with 256GB system RAM and roughly 400GB total VRAM; the user notes llama.cpp on many GPUs was very slow, possibly due to ROCm. The user also runs MiniMax M2.7 (MXFP4_16, iq4_nl) on 8x R9700 and Qwen3-27b-BF16 on 4x7900xtx under vLLM, and gave up on a REAP GLM-5.1 50% build due to vLLM errors.

generation:

13.6 tokens/s

prompt processing (prefill):

338.4 tokens/s

quant:

MXFP4 (GGUF)

User reports DeepSeek-V4-Flash at 13.61 t/s generation and 338.41 t/s prompt eval on 2 x RTX PRO 4000 with 48 GiB VRAM and 160 GiB system RAM. Setup is llama.cpp with an MXFP4/Q8_0 GGUF (about 146 GiB) at 32768 context, using --fit on and --no-mmap. A llama-batched-bench sweep from 8K to 1M context shows prompt throughput falling from 448.57 t/s to 65.97 t/s and generation from 19.20 t/s to 6.56 t/s; without hc-ops the same sweep was slower (322.50 t/s prompt, 14.62 t/s generation at 8K). KLD against the base logits is 0.422277 mean for the Q2 quant and 0.115089 for the Q4 quant.

generation:

14.9 tokens/s

prompt processing (prefill):

445.1 tokens/s

quant:

MXFP4 (MLX)

User reports DeepSeek-V4-Flash-0731-MXFP4-MLX decode at 14.89 tok/s on an M3 Ultra 512GB, about 2.8x below the 41.7 tok/s the model card reports for this checkpoint at 23K context with native DSpark MTP. Setup is oMLX 0.6.4 with MXFP4 MLX weights, native MTP enabled with 3 draft tokens, 23,716-token prompt, no KV cache quant stated. MTP acceptance is 86-88% and prefill is 445.08 tok/s, but decode does not follow. User notes the same ~14-15 tok/s class on 0.6.3rc3 and 0.6.4, no swap or memory pressure, and that 0.5.4rc2 (the reference build) was not A/B tested.

generation:

36.1 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

MXFP4

User reports DeepSeek V4 Flash MXFP4 at 36.1 t/s decode and 497 t/s prefill served across a Radeon 8060S iGPU and an RTX PRO 6000 Blackwell. Setup is llama.cpp with the rocm-cuda backend on two GPUs, the Strix Halo iGPU plus an external RTX PRO 6000 96 GB, MXFP4 weights (146 GB) fully GPU-resident with no CPU offload and the experts of 18 layers on the iGPU, under Windows. The repo also reports that on Blackwell a CUDA 13.3 build of llama.cpp loses 50-80% of decode past 8192 context: an unnamed 27B model on the RTX PRO 6000 drops from 72 to 35 t/s, while a CUDA 12.8 build of the same commit keeps 74.6 t/s at 8K. gpt-oss-120b runs 52 t/s on the iGPU alone, and Qwen3.8-Flash-Next Q4_K_M across two AMD GPUs gave 21.5-25.8 t/s decode at 4K.

generation:

13.9 tokens/s

prompt processing (prefill):

539.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

User reports DeepSeek V4 Flash 162B (REAP K144 pruned) at 13.899 tok/s decode and 538.958 tok/s prefill on a single DGX Spark. Setup is vLLM with NVFP4/MXFP4 expert weights and FP8 KV cache, 200K context, 14 GB KV cache, no speculative decoding. Validation used a 186,390-token prompt with 345.834 s TTFT and needle retained; the watchdog logged a low-memory kill at teardown, so the profile is proof-of-concept. K160 prefills at about 514 tok/s and decodes at about 24 tok/s with MTP.

generation:

9.8 tokens/s

prompt processing (prefill):

500.0 tokens/s

quant:

2-bit expert planes

User reports DeepSeek-V4-Flash-0731 at ~9.8 tok/s decode on a single DGX Spark (GB10, 128GB unified memory). Setup is a vLLM-based stack (vLLM-Moet adapted for unified memory and sm_121) with 2-bit expert planes and an FP4 tier, serving 256K context with a KV pool of ~660K tokens. Decode uses the built-in DSpark speculative head at k=2; the recommended k=7 was slower at 5.1 tok/s. Prefill is ~500 tok/s chunked at 2048, with a 28K-token prompt giving ~53 s TTFT. First boot took ~35 min for one-time expert quantization, later boots ~10 min via persistent cache. Tool calling and per-request thinking toggle work out of the box.

generation:

80.1 tokens/s

quant:

NVFP4 (NVFP4)

kv:

NVFP4

User reports DeepSeek-V4-Flash-Vision-Exp at 80.1 tok/s on count-to-100, 51.8 tok/s on code and 33.2 tok/s on prose, on 2x DGX Spark at TP2. Setup is vLLM with NVFP4 weights and nvfp4_ds_mla KV cache, 1M context, DSpark speculative decoding at k=5, KV pool 2.79M tokens / 19.12 GiB at gpu_memory_utilization 0.85. Without the Patch 4 shared-expert fix the same workloads measure 50.7, 47.2 and 30.4 tok/s. The 0731 text checkpoint reaches 78.4 tok/s peak at 98.9% acceptance with Patch 4, versus 66.1 peak and 60.2% acceptance without. The user notes a ~30% cold-start penalty and that realistic mixed agent traffic at concurrency 4 gives ~88.6 tok/s aggregate and ~22.3 tok/s per stream.

generation:

41.0 tokens/s

prompt processing (prefill):

1785.0 tokens/s

quant:

FP8 (FP8)

User reports DeepSeek-V4-Flash at ~41 tok/s single-stream and ~350 tok/s aggregate at c=32 on 2x DGX Spark (GB10, 128 GB unified) via vLLM TP=2 over QSFP56 RoCE. Setup is vLLM with official FP8 weights, 1M context profile, shared ~1.1M-token KV pool; prefill ~1785 t/s. Long-context sweep: 8k 40 t/s, 100k 40 t/s, 200k 38 t/s, 500k 32 t/s decode; prefill ~1970 down to ~1380 t/s. Cross-machine: RTX PRO 6000 ds4 46.9 t/s, M2 Ultra 29.7 t/s, single Spark ds4 IQ2_XXS ~14 t/s. llama.cpp GGUF suite up to 67 t/s (35B-A3B), 172B MoE 31 t/s.

generation:

28.0 tokens/s

quant:

int4 (safetensors)

kv:

fp8

User reports DeepSeek-V4-Flash at ~28 tok/s single-stream decode on one RTX 5090 with dual Xeon Gold 6138 and 256GB DDR4-2400. Setup is stock ktransformers + sglang with int4 AMXINT4 CPU-offloaded experts, NUMA-sharded, top-4 adaptive-k, 8192 context, fp8 KV cache. The 284B-A13B model is 142 GB at 4-bit and does not fit the GPU; experts are computed on the CPU. Re-verified at 27.9-28.0 tok/s steady-state decode. Decode is DDR4-bandwidth-saturated. EAGLE speculative decode (27.2), more GPU experts (27.5), and 80 hyperthreads (2.7) all lost. Coding canary passes 8/8.