DeepSeek V4

generation:

15.0 tokens/s

prompt processing (prefill):

138.0 tokens/s

quant:

IQ2XXS/Q4_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 at 15 t/s generation and 138 t/s prefill on an AMD Radeon 8060S. Setup is ds4-bench on ROCm 7.2.4 with a Q4K/IQ2XXS hybrid quant and a 65665-token context allocation. The figures come from the first 2048-token frontier of a 32-point sweep to 65536, with 2048 prefill and 128 generation tokens per row; generation falls to 12.17 t/s at the deepest frontier.

generation:

15.3 tokens/s

prompt processing (prefill):

152.2 tokens/s

quant:

IQ2XXS/Q2_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 at 15.3 t/s generation and 152.2 t/s prompt processing on an AMD Radeon 8060S. Setup is native ds4 on ROCm 7.2.4 with an IQ2XXS/Q2_K imatrix GGUF, ctx=65665, prefill_chunk=8192, and no speculative decoding. The headline figures come from the first 2,048-token frontier of a 32-point sweep from 2K to 64K, where generation falls from 15.25 t/s to 12.36 t/s. Each row times 2,048 prefill tokens and 128 generated tokens; the 32 frontiers are not independent trials.

User compares a baseline of Claude Code with DeepSeek V4 against an Attemory retrieval-hint setup. The Attemory setup reports a 43.8% token drop and a judge score tie, 83.39 versus 83.17, on SWE-QA. No hardware or inference engine details are given.

generation:

16.0 tokens/s

quant:

IQ3_XXS

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a patched engine improved speed from 5-6 to 15-16 tok/s. Setup uses the Unsloth UD-IQ3_XXS quant. The user notes a wired limit at 120GB.

generation:

15.0 tokens/s

prompt processing (prefill):

300.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 1M lossless context on 3x RTX 3090 with DDR5, at 15 t/s on cold start and 10 t/s steady. Setup is a llama.cpp fork with deepseek4 architecture support. A fourth RTX 3090 runs gemma 12b and flux2klein. Prefill starts at 300 t/s and degrades to 100 t/s after 64,000 tokens.

generation:

9.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 running at 9 t/s on Windows. The system went to sleep despite the user's settings, causing hardware failure. No hardware details are provided.

prompt processing (prefill):

255.0 tokens/s

quant:

W8A8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports custom Turing CUDA kernels for W8A8 INT8 matmul, heterogeneous inference with 4x 11/22GB VRAM and 1TB system RAM, and computation-communication overlap for MoE routing. The work is open-sourced on GitHub.

generation:

51.1 tokens/s

quant:

q2

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a Lucebox custom setup with an AMD Radeon AI PRO R9700 and Strix Halo 128GB running 3.63x faster than a single DGX Spark. Setup uses asymmetric parallelism: the R9700 handles the dense path, hot experts, cache and draft model, while the Strix Halo holds the other experts. The run is experimental, at q2 quant and 16k context. The user is working on KVFlash for 64k-128k.

User is setting up a 16x DGX Spark cluster to run frontier models locally. User mentions DeepSeek V4 pro, Kimi K3, GLM 5.5, and Minimax M4 as future models. No benchmark numbers are provided.