DeepSeek R1

generation:

9.9 tokens/s

quant:

Q2_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks DeepSeek R1 671B at 9.9 t/s generation on a dual-socket Intel Xeon 6980P server with 1.5 TB RAM. Setup is ik_llama.cpp with Q2_K_XL GGUF quant and F16 KV cache, running CPU-only on a single NUMA node with 43 threads. The user compares llama.cpp (8.9 t/s) and ik_llama.cpp (9.9 t/s) for Q2_K_XL, and also tests Q4_K_M (10.0 t/s) and Q8_0 (7.5 t/s) with ik_llama.cpp. Dual-socket configurations degraded performance.

generation:

4.2 tokens/s

quant:

Q5_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-R1 671B at 4.2 t/s on a single AMD EPYC 7k62 (48 cores) with 8x64GB RAM, CPU only. Setup is llama.cpp with Q5_K_S GGUF weights (461.81GB). User also tested an AMD Threadripper Pro 3955WX (16 cores, 8x64GB RAM) at 2.8 t/s and dual AMD EPYC 7k62 (96 cores, 16x64GB RAM) at 2.9 t/s, and concludes llama.cpp is not optimized for dual-socket motherboards. User suggests a single-socket EPYC 9654 with 12 DDR5 channels and expects around 10 t/s.

generation:

2.0 tokens/s

quant:

Q3 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 2 t/s running DeepSeek-R1-0528 Q3 GGUF on 2x RTX 3090 with 48 GB VRAM and 512 GB DDR5 system RAM. No engine is named. The user asks for upgrade advice to reach 15+ t/s with a very large model, and asks about Threadripper, EPYC and Xeon platforms.