Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports about 2 t/s running DeepSeek-R1-0528 Q3 GGUF on 2x RTX 3090 with 48 GB VRAM and 512 GB DDR5 system RAM.
No engine is named.
The user asks for upgrade advice to reach 15+ t/s with a very large model, and asks about Threadripper, EPYC and Xeon platforms.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarizationagentic
User compares ninfer-3090 against llama.cpp on an RTX 3090 for Qwen3.6-35B-A3B and Qwen3.8-27B.
The primary record is Qwen3.6-35B-A3B with ninfer-3090. llama.cpp with the UD-IQ4_XS GGUF reached 2439.7 t/s prefill, 148.6 t/s generation and 3410 ms TTFT. For Qwen3.8-27B, ninfer reached 187911.6 t/s prefill, 34.9 t/s generation and 27 ms TTFT, while llama.cpp reached 1012.7 t/s prefill, 38.5 t/s generation and 8160 ms TTFT.
The benchmark used 7 prompts (4 long at roughly 7.7k-12.9k tokens, 1 medium at roughly 2.3k, 2 short at roughly 400-600), 3 repeats each, max_tokens=1024, seed=42 and cache busting via nonce. The ninfer-3090 system is a Ryzen 5 2600 with 16 GB RAM; the llama.cpp system is a Threadripper PRO 3945WX with 128 GB RAM. The user notes the hardware is not apples-to-apples but that inference is GPU-contained.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports token generation improving from 20 t/s to 49 t/s with the flashnext-e06 fork on 2x RTX 3090, an Intel Ultra 7 270k Plus, and 192 GB DDR5@5600 MHz.
Setup uses an MTP draft model in Q8_0 and ngram-mod speculative decoding.
Prompt processing is faster on the main branch.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports prefill improvements of 2.2-2.5x on Qwen3.8-Flash-Next across 8k, 37k, and 119k contexts on 2x RTX 3090 with dual Broadwell Xeon and DDR4, with decode performance unchanged.
Setup is llama.cpp on a custom branch flashnext-e06, a 150-slot expert cache moved off GPU during prompt processing, 261k context, and an f16 KV cache. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC.
Prefill is 223.7 t/s at 8k (was 99.9), 212.6 t/s at 37k (was 88.1), and 206.5 t/s at 119k (was 81.3). Decode is 34.3 t/s at 8k, 41.2 t/s at 37k, and 33.9 t/s at 119k. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, and 1461s to 575s at 119k. A quality screen showed no regression, and MTP acceptance is 0.79-0.83.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Huihui Qwen3.8-27B Abliterated Q4_K_S at an average TG of ~25.58 tok/s on an RTX 3090 24GB at 131072 context, in a conversation of 111797 tokens.
Setup uses a Q4 KV cache with 23.5/24.0 GB dedicated and 0.6/31.8 GB shared GPU memory, on a Xeon E5-2670 v3 with 48GB DDR4-2133, and each run logs a DTA of 64.2% to 70.8%.
Individual runs read 22.02 to 31.94 tok/s. At 196608 context the average TG is 25.80 tok/s with an average DTA of 64.57%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 285B MoE at 60+ t/s on 10 RTX 3090s and 120+ t/s on 12 RTX 3090s.
Setup is FP4 experts with FP8 attention and 157GB of weights, using TP2xPP5 on 10 GPUs at a 240W cap and TP4xPP3 on 12 GPUs.
The 60+ t/s figure is on 10 GPUs. Context runs to 1M without offload and 4M with RAM offload, with long-context prefill around 3500 t/s when batched. Vision, speculative decoding and tool calls work. A Docker image and repo are provided.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User reports median decode improving from 30.2 to 33.3 t/s with radix-selection top-k fallback.
A quality screen scored 238/240 correct against 235/240 for the control, with no regression detected.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-contextagentic
User reports Muse Glimmer fits on a single RTX 3090 at Q4_K_XL with full context, DFlash and mmproj, using about 22-23 GB of VRAM.
Generation speed ranges from 64 to 124 t/s depending on prose versus code, with an average of 94 t/s. Prompt processing runs at about 1400 t/s.
The user notes Qwen3.6-27B and Gemma-4-31B are borderline unusable on the same GPU.
User reports gpt-oss-20b at close to 200 tps aggregate across parallel requests with the full 128k context on a single RTX 3090, under a custom harness (burrito) that fixes the model's tool calling; 320,192 evals over 8 seeds and 3.49B tokens took 1,062 GPU hours at batch size 1.
Setup is factory-precision gpt-oss-20b weights on one RTX 3090 serving parallel requests; no single-stream figure and no decoding method is given.
Effort levels affect accuracy: Low 38.3%, Medium 97.1%, High 100.0%. The user mentions llama.cpp and vLLM had issues with the model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-5.2 UD-IQ2_M at ~7.3 tok/s decode and 135 tok/s prefill on 4x RTX 3090 (1 Ti + 3 FE), 96GB total VRAM with only ~83GB used.
Setup is llama.cpp master with the glm-dsa arch, a 223GB on-disk model at 744B total / 40B active, expert offload to CPU, explicit expert placement via -ot to avoid OOM, and a ~6GB KV cache at 128K that scales linearly.
Decode is bound by CPU compute, not bandwidth. IQ1_M and IQ2_M are identical in A/B tests (7.30 vs 7.29 tok/s), and 6 vs 12 CPU threads gave +22% (5.83 vs 7.14 tok/s). Thinking off gives ~5-6x wall-clock speedup.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User compares reported token rates across several local rigs, with Qwen3.6 27B on a single RTX 3090 at about 50 tok/s generation and about 950 tok/s prompt processing.
That rig runs a Q4/Q8 MTP configuration at 128k context.
The figures are collected second-hand from a discord group rather than measured by the user, who also lists about 140 tok/s on a 5090 and 80-100 tok/s for DeepSeek V4 Flash on dual RTX 6000 cards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports DeepSeek V4 Flash at 37.47 t/s pp and 6.62 t/s tg on a dual Xeon E5-2660 v2 with an RTX 3090.
Setup is the Q4_K_XL quant at 8k context.
The user also tested Cydonia v4.3, a Mistral Small 3.2 finetune, at 308.92 t/s pp and 7.8 t/s tg.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks DeepSeek V4 Flash GGUF in IQ3_XXS-AS and IQ2_S quants on an RTX 3090 with 128 GB DDR4.
Setup is mainline llama.cpp b10064, which matches the fairydreaming fork's performance.
User prefers IQ3_XXS-AS as the sweet spot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports DeepSeek V4 Flash at 37.91 t/s prompt processing and 9.07 t/s generation on two RTX 3090 24GB cards and one RTX 5090 32GB card, 80GB VRAM total, with 128GB DDR4 RAM.
Setup is llama.cpp server with the unsloth Q8_K_XL GGUF, tensor splitting, and MoE layers offloaded to CPU.
User says performance is better than Qwen 3.5 v2 on coding tasks and asks for speed improvement suggestions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports llama-bench results on 3x RTX 3090 with Q3_K_XL quantization.
The model is 119.40 GiB with 284.33 B params.
Prompt processing runs at 116.04 t/s and generation at 7.71 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 UD-IQ3_S running under the llama.cpp loader in text-generation-webui.
Setup uses --n-cpu-moe 39 to offload MoE experts to system RAM, with the loader estimating roughly 136 GB total and relying on 128 GB of DDR5 at 5600 MHz.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User requantized only the 129 routed expert tensors of DeepSeek-V4-Flash-0731 to IQ3_XXS, with down projections at IQ3_S, leaving attention, shared experts and router at Q8_0, BF16 and F32. The result is 111.37 GiB in four shards.
Against the MXFP4 source, KLD is a mean of 0.2386 versus 0.2936 for UD-IQ3_S, top-1 is 84.65% versus 82.78%, and delta PPL is +0.536 versus +0.685, but the requant is 2.12 GiB larger and has a worse max KLD of 12.53 versus 11.13.
On 5 mixed GPUs (2x 3090, 5060 Ti, 2x 4060 Ti, 96 GiB VRAM) with expert spill to CPU, speed is 13.91/13.57/13.26 t/s at depths 0/4096/16384 versus 9.88/9.69/9.51 for the MXFP4 source, roughly 1.4x. antirez's flat Q2 is 80.76 GiB, about 98% resident, at 30.27 t/s. The user notes known DSV4 SWA/rollback stall issues in llama.cpp and used a patched build.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 1M lossless context on 3x RTX 3090 with DDR5, at 15 t/s on cold start and 10 t/s steady.
Setup is a llama.cpp fork with deepseek4 architecture support. A fourth RTX 3090 runs gemma 12b and flux2klein.
Prefill starts at 300 t/s and degrades to 100 t/s after 64,000 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at ~9.9 tok/s with the UD-IQ4_XS quant on a single RTX 3090 at 128K context.
Setup is llama.cpp b10291 with Text Generation WebUI, Q8_0 KV cache, 44 GPU layers and --n-cpu-moe 38 keeping expert tensors in 128 GB of DDR5-5600 on a Ryzen 9 9900X.
Smaller quants run faster: UD-IQ3_S at ~12.1 tok/s, UD-IQ3_XXS at ~12.5 tok/s and UD-IQ2_M at ~14.9 tok/s.
User compares a W4A16 quant of Nemotron 3.5 Lightning 30B-A3B against an IQ4_XS GGUF on an RTX 3090.
Setup is vLLM for W4A16 and llama.cpp for IQ4_XS.
The two are near-parity in instruction following benchmarks, with about 4.5x throughput by B16. The user describes the model as fast and reliable, and suitable for batch labelling and agentic responses.