Local model performance on your hardware

generation:

79.0 tokens/s

quant:

2.875 bpw

User reports DeepSeek-V4-Flash-0731 at 79 tok/s greedy decode on four Radeon AI PRO R9700 cards. Setup is the affinity engine with a prebuilt 2.875 bpw quant, all 11,008 experts resident in VRAM with no host pool allocated, on a Threadripper PRO 9955WX with 128 GB RAM. The same box on two cards gave 36 tok/s with 45% of experts in VRAM and the rest streaming from a 51 GB pinned host pool. Sampled decode at temp 1.0 was 63 tok/s and about 90 tok/s on a chat answer with thinking on. Load takes about a minute, 42 seconds of it pushing 96 GB of experts onto the cards.

generation:

19.9 tokens/s

prompt processing (prefill):

270.1 tokens/s

quant:

Q4_K experts / mixed F16-Q8 (GGUF)

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash at 19.9 t/s generation and 270.1 t/s prompt processing on two AMD Strix Halo nodes with 128 GB RAM each, linked by OdinLink USB4/TB5 RDMA. Setup is the Antirez Q4_K experts build with F16 HC/compressor/indexer and Q8 attention/shared/output, tensor parallel across two Radeon 8060S iGPUs with a balanced 50/50 expert split. The run used a fixed 2,048-token prompt followed by 300 generated tokens, with ordinary decode and no speculative decoding; the source gives 19.89 t/s and 270.10 t/s exactly.

generation:

18.6 tokens/s

prompt processing (prefill):

140.3 tokens/s

quant:

IQ3_XXS (GGUF)

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash IQ3_XXS at 18.6 t/s generation and 140.3 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 434ddbb with Vulkan on a single Radeon 8060S, 120GB GTT region, and a 97.05GiB tensor size. The figures are separate PP512 and TG128 synthetic tests, not a 512-input/128-output client request; the source gives 18.56 +/- 0.01 t/s and 140.30 +/- 4.35 t/s over 3 repeats.

generation:

17.3 tokens/s

prompt processing (prefill):

254.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

Q8

flash attention:

on

User reports DeepSeek V4 Flash at 17.3 t/s generation and 254.3 t/s prompt processing on an AMD Ryzen AI MAX+ 395 with Radeon 8060S and 128 GB unified memory. Setup is llama.cpp (Nathanw1014/strix-halo-llamacpp v0.7.5, Vulkan backend) with UD-IQ3_XXS, K/V q8_0, flash attention, and no speculative decoding, on one Strix Halo GPU. The headline figures are the 7,680-input point of a three-point curve; at 30,720 input the rates are 247.7/17.4 and at 122,879 input 194.8/15.8 t/s.

generation:

28.5 tokens/s

prompt processing (prefill):

241.1 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

Q8

flash attention:

on

User reports DeepSeek V4 Flash at 28.5 t/s generation and 241.1 t/s prompt processing on an AMD Ryzen AI MAX+ 395 with Radeon 8060S graphics and 128 GB unified memory. Setup is llama.cpp (Nathanw1014/strix-halo-llamacpp v0.7.5, Vulkan backend) with UD-IQ3_XXS, K/V q8_0, flash attention and DSpark fixed draft length 4, at 7,680 input tokens. The same run measured 32.28 t/s at 30,720 input tokens and 28.11 t/s at 122,879 input tokens; the headline figures are the 7,680-input point, not a maximum across contexts.

generation:

21.2 tokens/s

prompt processing (prefill):

321.5 tokens/s

quant:

ROCmFPX-MIX-STRIX (GGUF)

User reports DeepSeek V4 Flash at 21.2 t/s generation and 321.5 t/s prompt processing on one AMD Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151) and 128 GB unified memory. Setup is Lucebox PR 729 on ROCm/HIP 7.2.4 with ROCmFPX-MIX-STRIX, K/V q4_0, fused decode and verify with F16 KV, expert top-k 6, sparse prefill, and a 131,072-token allocation; speculation is disabled with no drafter loaded. The headline figures are the 7,680-input point of a native PP/TG curve, not a maximum across contexts; the same curve gives 318.0/18.4 at 30,720 input and 287.4/12.9 at 122,879 input.

generation:

41.9 tokens/s

prompt processing (prefill):

283.3 tokens/s

quant:

ROCmFPX-MIX-STRIX (GGUF)

User reports DeepSeek V4 Flash at 41.9 t/s generation and 283.3 t/s prompt processing on an AMD Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151) and 128 GB unified memory, one Strix Halo GPU. Setup is Lucebox PR 729 on ROCm/HIP 7.2.4 with ROCmFPX-MIX-STRIX, K/V q4_0, fused decode and verify with F16 KV, expert top-k 6, sparse prefill, and DSpark adaptive q2-q5 speculation, at a 131,072-token allocation. The headline figures are the 7,680-input point of a native PP/TG curve, not a maximum across contexts; the same curve gives 306.6/41.4 at 30,720 input and 283.9/38.0 at 122,879 input.

generation:

19.4 tokens/s

prompt processing (prefill):

52.2 tokens/s

quant:

MXFP4 (GGUF)

kv:

F16

flash attention:

on

User reports DeepSeek V4 Flash 0731 at 19.4 t/s generation and 52.2 t/s prompt processing on one RTX PRO 5000 Blackwell 72 GB plus one RTX PRO 5000 Blackwell 48 GB. Setup is llama.cpp CUDA b11063 with MXFP4, f16 K/V cache, flash attention, layer split 4.5/2 and -ncmoe 11, leaving about 110.6 GiB of weights on the GPUs. The 52.2 t/s is the PP512 cell and the 19.4 t/s is the separate TG128 cell, one repetition each; the 145.63 GiB GGUF exceeds the 118.39 GiB combined VRAM.

prompt processing (prefill):

454.6 tokens/s

quant:

IQ2_XXS + Q2_K, selected Q8 tensors (GGUF)

User reports DeepSeek V4 Flash prefill at 454.6 t/s on an AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151) with 128 GB unified memory. Setup is the DS4 Halo ROCm prefill fork with IQ2_XXS + Q2_K and selected Q8 tensors, 4,352-token context allocation and 4,096-token prefill chunks. The figure is a warmed prepared-prefill mean of 454.59 t/s from two samples, 454.52 and 454.65, on a complete position-zero 4,096-token request with generation disabled.

generation:

17.8 tokens/s

prompt processing (prefill):

238.0 tokens/s

quant:

UD-IQ3_S

mtp (multi-token prediction):

off

User reports DeepSeek V4 Flash 284B-A13B at 17.8 t/s generation and 238 t/s prompt processing on an AMD Gorgon Halo 192GB machine. Setup is llama.cpp with UD-IQ3_S, Vulkan backend, all layers on GPU, on a Minisforum MS-S1 MAX-P495 with Ryzen AI Max+ PRO 495 and 192 GB unified memory. The 238 t/s and 17.8 t/s come from separate llama-bench pp512 and tg128 tests, not a paired 512-input/128-output conversation; the same source reports roughly 32k-depth prompt processing at 106 t/s and generation at 13.2 t/s.

generation:

15.4 tokens/s

prompt processing (prefill):

142.3 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

User reports DeepSeek-V4-Flash at 15.4 tok/s decode and 142.30 tok/s prefill at 2,048 tokens of context on a Ryzen AI Max+ PRO 395 (Strix Halo) with 128 GB unified memory. Setup is the ds4 engine with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF (81 GB), 32,768-token context, ROCm 7.1.1; decode falls 12% from 2K to 32K while prefill stays flat. A second model, Qwen3.8-Flash-Next UD-IQ4_XS (88 GB) under Strata 0.1.40, decodes at 39.0 to 40.9 tok/s and prefills a 6,786-token prompt at 343 to 367 tok/s, dropping to 30.4 tok/s each with two requests in parallel. The same engine measured 8.7 tok/s while another model's worker held 41 GiB of VRAM and 23 GB of RAM, against 39.0 tok/s on an exclusive box.

generation:

31.7 tokens/s

quant:

IQ2_XXS/Q2_K (GGUF)

User reports DeepSeek V4 Flash Vision-Exp at 31.7 tok/s decode on a Mac Studio M4 Max 128GB. Setup is ds4 (DwarfStar) with IQ2_XXS/Q2_K GGUF, DSpark speculative decoding at confidence 0.6, and a 32768 context length. Thinking-on pagoda run reached 29.84 tok/s over 26,415 completion tokens; 1M context loaded at 97.15 GiB planned total with 32.0 tok/s decode. User notes DSpark adds roughly 10-20% over no speculation.

generation:

36.1 tokens/s

prompt processing (prefill):

498.3 tokens/s

quant:

2.44bit-mixed (MLX)

User reports DeepSeek V4 Flash 0731 at 36.1 t/s generation and 498.3 t/s prefill on a MacBook Pro M5 Max 128GB 40 GPU. Setup is oMLX 0.5.7 with a 2.44 bpw mixed MLX quantization (92.8 GB on disk) at 1k prompt, single request, 128 generated tokens, speculative decoding off. Generation holds 33.8 t/s at 4k and 31.5 t/s at 32k, with peak memory rising 1.03 GB across an 8x context jump; prefill at 32k takes 97 seconds to first token. Continuous batching reaches 83.2 t/s at batch 8 (2.30x).

generation:

26-29 tokens/s

prompt processing (prefill):

542.0 tokens/s

quant:

mxfp4 (MLX)

User reports DeepSeek-V4-Flash-0731 at 542 tok/s prefill on 64k context on 2x Mac Studio 512GB. Setup is MLX with mxfp4 routed experts and affine-8 gs64 elsewhere (155 GB), tensor-parallel over Thunderbolt RDMA, 2 nodes. Decode is ~26-29 tok/s flat at 64-256k context and ~31 tok/s flat at 6k generated tokens, up from ~10 and 17.7 decaying on the stock baseline. Prefill reaches 510 tok/s at 128k and 366 tok/s at 256k, with 128k prefill transient memory cut from +99 GB to ~24 GB per node. Recall is 5/5 at all depths to 256k. User notes 512k+ cold prefill still exceeds the per-node transient budget beside a resident second model.

generation:

45.9 tokens/s

prompt processing (prefill):

362.3 tokens/s

quant:

q2-imatrix (GGUF)

User reports DeepSeek-V4-Flash q2-imatrix at 45.9 t/s generation and 362.3 t/s prefill on an RTX Pro 6000 Blackwell 96GB at 2048 context. Setup is ds4 with q2-imatrix GGUF, fp16 dequant cache disabled and MTP off, fully resident at ~91 GB peak. The 80.76 GB model only fits with DS4_CUDA_Q8_F16_CACHE_MB=0; MTP adds 3.5 GB and pushes over 96 GB. Generation beats posted M5 Max and DGX Spark GB10 q2 numbers; prefill is about equal to Spark. At 28672 context generation drops to 40.6 t/s. The user notes the card is tight for this model and heavier ds4_test sessions OOM.

generation:

28.9 tokens/s

prompt processing (prefill):

261.7 tokens/s

quant:

UD-IQ2_XXS (GGUF)

User reports DeepSeek-V4-Flash-0731 (284B MoE, 13B active) at 28.91 t/s generation and 261.7 t/s prompt processing on a single AMD Ryzen AI MAX+ 395 with 128GB unified memory. Setup is a llama.cpp fork (Nathanw1014/llama.cpp) with Vulkan/RADV backend, UD-IQ2_XXS quant (2.06 bpw), DSpark draft model at n=3, 16384 context, 120 GiB GTT limit. Best result is IQ2_XXS + DSpark n=3 at 28.91 t/s, reaching 90% of Lucebox's 32.0 t/s with 6 experts instead of 4. The fork beats upstream by 49% on prompt processing at 8192 context. IQ2 with draft model sits at the memory limit and is a measurement setup, not a production state.

generation:

16.3 tokens/s

prompt processing (prefill):

484.6 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

User reports DeepSeek V4 Flash at 484.62 t/s prefill and 16.28 t/s decode on a Strix Halo 128GB (Ryzen AI MAX+ 395, Radeon 8060S, gfx1151). Setup is Gufo with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 mixed GGUF, one session, speculative decoding disabled, thinking off, temperature 0, seed 1, nominal pp2048/tg128 (2,035 prompt tokens, 128 generated). One warmed sample; no standard deviation reported. The maintainer notes target-model quality parity is unestablished and build-sensitive differences are under investigation. Power, KV-cache type, Flash Attention and peak memory were not reported.

generation:

40.0 tokens/s

User reports DeepSeek V4 Flash reaching a peak of 40 t/s generation on 8x Radeon AI PRO R9700 under llama.cpp, but the run froze after 200-300 tokens. Setup is llama.cpp on 8x R9700 with 256GB system RAM and roughly 400GB total VRAM; the user notes llama.cpp on many GPUs was very slow, possibly due to ROCm. The user also runs MiniMax M2.7 (MXFP4_16, iq4_nl) on 8x R9700 and Qwen3-27b-BF16 on 4x7900xtx under vLLM, and gave up on a REAP GLM-5.1 50% build due to vLLM errors.

generation:

13.6 tokens/s

prompt processing (prefill):

338.4 tokens/s

quant:

MXFP4 (GGUF)

User reports DeepSeek-V4-Flash at 13.61 t/s generation and 338.41 t/s prompt eval on 2 x RTX PRO 4000 with 48 GiB VRAM and 160 GiB system RAM. Setup is llama.cpp with an MXFP4/Q8_0 GGUF (about 146 GiB) at 32768 context, using --fit on and --no-mmap. A llama-batched-bench sweep from 8K to 1M context shows prompt throughput falling from 448.57 t/s to 65.97 t/s and generation from 19.20 t/s to 6.56 t/s; without hc-ops the same sweep was slower (322.50 t/s prompt, 14.62 t/s generation at 8K). KLD against the base logits is 0.422277 mean for the Q2 quant and 0.115089 for the Q4 quant.

generation:

14.9 tokens/s

prompt processing (prefill):

445.1 tokens/s

quant:

MXFP4 (MLX)

User reports DeepSeek-V4-Flash-0731-MXFP4-MLX decode at 14.89 tok/s on an M3 Ultra 512GB, about 2.8x below the 41.7 tok/s the model card reports for this checkpoint at 23K context with native DSpark MTP. Setup is oMLX 0.6.4 with MXFP4 MLX weights, native MTP enabled with 3 draft tokens, 23,716-token prompt, no KV cache quant stated. MTP acceptance is 86-88% and prefill is 445.08 tok/s, but decode does not follow. User notes the same ~14-15 tok/s class on 0.6.3rc3 and 0.6.4, no swap or memory pressure, and that 0.5.4rc2 (the reference build) was not A/B tested.