NVIDIA RTX 3090 reports

generation:

94.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3-Next-80B-A3B at 94.5 tok/s decode on an RTX 3090 with 16 GB RAM, using a patched llama.cpp that substitutes missing experts instead of waiting for SSD reads. Setup is llama.cpp with Q4_K_M, 1/4 of experts in VRAM, rest read from NVMe at ~5.7 GB/s, 16 threads, about 15.7 GB VRAM used. Stock llama.cpp gave 31.8 tok/s in the same 16 GB case; the patch also reached 108.4 tok/s with plenty of RAM and 89 tok/s reading every miss from SSD. Perplexity was 1.6% higher than stock, GSM8K lost 1.8 points, and greedy generation ran 64-74 tok/s. The 16 GB case was simulated by locking RAM on a bigger machine.

generation:

116.0 tokens/s

prompt processing (prefill):

2908.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 116 t/s decode and 2,908 t/s prompt read on two RTX 3090s with NVLink. Setup is a modified Strata build (v0.1.38 plus 29 commits) with IQ3_S GGUF at 262K context, greedy, median of 3, on a Ryzen 9 3950X with 121 GB RAM. The second card acts as a peer expert tier read over NVLink; the pair holds about 19,500 of 24,576 IQ3_S experts with a 0.98 hit rate on real agent chats. Decode gains from the second card are modest (+10%) because the verify window is latency-bound on the primary. Sampled with official settings it sits around 102-104 t/s, a 1.5K prompt reads at ~1,670 t/s, and switching chats takes 1 s. One stream at a time; answers are word-for-word identical to the one-card run.

generation:

48.8 tokens/s

prompt processing (prefill):

142.5 tokens/s

quant:

Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B Q8_K_XL at 48.76 t/s generation and 142.46 t/s prompt processing on dual RTX 3090s. Setup is llama.cpp via llama-swap with MTP speculative decoding (draft-mtp, n-max 3), 238000 token context, tensor split 50,50 across two cards, batch size 4096. The figures are from the final turn of a multi-turn agentic coding run (95 prompt tokens, 90 generated, 43473 cached); per-turn generation speeds ranged from 39.58 to 77.38 t/s across the run.

generation:

7.2 tokens/s

prompt processing (prefill):

111.1 tokens/s

quant:

Q8KXL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 294B Q8KXL at 7.2 t/s generation and 111.09 t/s prompt processing on a 10-GPU server (8x RTX 3090 + 2x RTX 5090, 256GB VRAM). Setup is koboldcpp with llama.cpp under the hood, GGUF weights, all 10 cards allocated with tensor split, clocks locked to 1200 MHz and power limited to 200W per 3090 to prevent transient spikes. The figures are from the first of seven large cybersecurity prompts (2769 input tokens, 813 output tokens); later prompts in the sweep drop to 4.28 t/s generation and 28.38 t/s prompt processing. The user notes the model was the smallest tested but punched above its weight in quality, and that the speed drop versus other models was an acceptable trade-off.

generation:

41.4 tokens/s

quant:

BF16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B BF16 at 41.4 tok/s decode on a single RTX 3090. Setup is Nano-vLLM with BF16 weights, CUDA Graph decode, batch 1, greedy decoding, 24 GB card at 0.9 GPU memory utilization. Decode is memory-bandwidth bound at 24.2 ms/token; eager mode gives about 17 tok/s and 4 concurrent requests give 138 tok/s aggregate. Prefill latency is 238 / 254 / 378 / 899 ms for 16 / 128 / 512 / 2048 tokens. The repo also reports Qwen3.6-27B-FP8 at about 41 tok/s on 4x RTX 4090 and Qwen3.5-9B at about 98 tok/s on 4x RTX 4090.

generation:

84.2-88.8 tokens/s

prompt processing (prefill):

3401-3413 tokens/s

quant:

W4A16 (safetensors)

kv:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on two RTX 3090s with 128 GB RAM, reaching 3,401–3,413 tok/s prefill and 84.2–88.8 tok/s decode on a 131,099-token prompt. Setup is vLLM with W4A16 AutoRound weights, BF16 KV cache, FP8 PLE table, MTP3 speculative decoding, and CPU expert offload. The 64 GB RAM profile runs both cards with 64 GB system RAM, keeping 88 experts per layer on each GPU and streaming the rest from RAM; prefill uses 8,192-token chunks. A single RTX 3090 with 64 GB RAM reaches 2,106 tok/s prefill and 47.4 tok/s decode at 131K context.

generation:

37-40 tokens/s

prompt processing (prefill):

1700.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports running DeepSeek V4 Flash with Strata on an RTX 3090 alone at 1700 t/s prefill and 37-40 t/s generation, and on both an RTX 3090 and RTX 5070 Ti at 1850 t/s prefill and 45-50 t/s generation, at 131K context. Setup is Strata with UD-Q4_K_XL quant, 96GB DDR4 system RAM, RTX 5070 Ti on PCIe x16 and RTX 3090 on PCIe x4. User compares against Unsloth Studio on the same model at 131K context, which gave 100 t/s prefill and 14-16 t/s generation, with a 100K conversation taking over 10 minutes to process. User notes Strata initially lacked prefix caching but now supports it, and that the same generation speed matches Qwen3.8-27B-UD-Q8_K_L on both GPUs.

generation:

96.2 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 96.2 tok/s on prose and 141.1 tok/s on code on an RTX 3090. Setup is SGLang with EXL3 3.0 bpw weights and an MTP draft model, single request. The figures come from the trellis-serve project's registry recipes; the same table also lists Qwen3.6-35B-A3B at 251.7 tok/s prose and 357.9 tok/s code on the RTX 3090, and Qwen3.8-27B at 91.2 tok/s prose and 66.1 tok/s code on an Intel Arc Pro B70.

generation:

36.5 tokens/s

quant:

Q4_K_S (GGUF)

kv:

q5_0/q4_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma-4-31B at 36.5 tok/s on a single RTX 3090 with full big context, using mainline llama.cpp built from source with -DGGML_CUDA_FA_ALL_QUANTS=ON. Setup is llama.cpp with Q4_K_S weights and q5_0/q4_1 KV cache on one 24 GB card; the stock prebuilt image only reaches ~12 tok/s because the flag is off by default. The post compares engines: beellama.cpp with DFlash speculative decoding reaches 54-193 tok/s depending on workload (131-200K context), vLLM gets 86-145 tok/s on dual cards at 262K, and ik_llama is fast but context-capped near 24K. Quality on an 8-pack was 109/150 with reasoning off and 114/150 with reasoning on.

generation:

38.1 tokens/s

prompt processing (prefill):

1263.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 38.1 t/s generation and 1263 t/s prefill on 2x RTX 3090, asking whether NVLink would improve performance over a better motherboard. Setup is llama.cpp b10586 (CUDA) behind llama-swap on Windows 11, with a Ryzen 7 5800X3D and 64GB DDR4. Cold load took 24s. At 28k context the run gave 1250 t/s prefill and 36.2 t/s generation; at 85k context 901 t/s prefill and 28.9 t/s generation. User also runs nightly data classification jobs and asks whether prefill speed would benefit.

generation:

34.0 tokens/s

prompt processing (prefill):

520.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at Q8 running at 34 t/s generation and up to 520 t/s prefill on 2x RTX 3090 with 256GB of system RAM. Setup uses the Strata engine with MTP set to 5; most of the model sits in system RAM rather than GPU memory. For comparison, the user says llama.cpp without MTP gives 13 t/s generation and 115 t/s prefill on the same setup, and notes MTP usually does not help when most of the model is in system RAM.

generation:

75.8 tokens/s

prompt processing (prefill):

569.2 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B dense at 75.81 tok/s decode on an RTX 3090 eGPU attached to a MacBook Air M2 over Thunderbolt 4. Setup is the ThunderLlamaX engine with IQ3_XXS GGUF weights, 100,352-token context, K=10 n-gram LOOKUP speculative decoding, and a custom DriverKit dext replacing the missing NVIDIA driver stack. Prefill is 569.2 tok/s at 2k, 510.1 at 8k, and 342.0 at 100k. On novel prose the EAGLE draft head gives 43.7 tok/s. A Qwen3.6-35B-A3B MoE model runs at 97.6-104.0 tok/s quote-class and 40.1 tok/s prose with 982.0 tok/s prefill at 2k. GSM8K is 95.0% dense and 93.0% MoE. A 100k context restores from cache in about 6.5 seconds.

generation:

169.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B-A3B-BigBang-MTP at 169.4 tok/s with MTP speculative decoding on a single RTX 3090, up 29% from 131.0 tok/s without the draft. Setup is llama.cpp with Q4_K_M main weights and a Q8_0 draft, single RTX 3090, temp 0.1. The grafted Qwen3.6 MTP head accepts 0.55-0.75 of tokens (code high, chat lower) with mean accepted runs of 3.2-4.0 tokens, versus ~0.13 acceptance for stock Ornith-1.5's random-init head. Stock Ornith-1.5 measured 133.9 tok/s without a draft. Perplexity on mixed code and prose was 3.34 versus 3.41 for stock, and both scored 15/15 on a 15-task Python pass@1. The author calls this a small local eval, not a benchmark suite.

generation:

80-110 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-110 tok/s on 2x RTX 3090 with 96GB DDR5 system RAM. Setup uses the Strata engine with a Q4_K_XL GGUF quant from bitlamas, at 2500 prompt tokens. The user calls it the biggest thing since Qwen 3.8 27B, offering the speed of Qwen 3.5 35B-A3B with higher intelligence than 3.8 27B.

generation:

136.8 tokens/s

prompt processing (prefill):

1581.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 136.8 tok/s single-stream decode (8k context, concurrency 1) on 4x RTX 3090. Setup is SGLang with AWQ-INT4 (W4A16 Marlin) weights and bf16 KV cache, tensor parallel 4, DSpark speculative decoding enabled with a 1.36B draft model, 275,018-token KV pool. Prefill sustains 1,581 tok/s at 8k c1 and 1.3-1.6k tok/s across all context lengths; aggregate decode reaches 169.8 tok/s at 8 concurrent requests. Max context 128k, max 9 concurrent requests (GDN state bound). Also benchmarks 2x 3090 (112.9 tok/s decode at 8k c1) and 1x 3090 without DSpark (43.2 tok/s decode, 8k context limit).

generation:

98.2 tokens/s

prompt processing (prefill):

914.0 tokens/s

quant:

3.00bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 98.2 t/s on prose and 225.1 t/s on code on one RTX 3090. Setup is SGLang with the sglang-exl3 plugin, EXL3 3.00bpw weights, fp8 KV cache, a 131,072-token window, and DFlash2 speculative decoding with a 5.0bpw draft. Prompt processing on a new uncached 32k prompt runs at about 914 tok/s with 35.9 s to first token. Compared with the MTP recipe on the same card, code is +57% (225 vs 143) and prose is the same (98 vs 98), with a 131k window instead of 262k. Max concurrency is 1 because two 32k streams stall each other's decode.

generation:

129.0 tokens/s

prompt processing (prefill):

786.0 tokens/s

quant:

exl3-3bpw (EXL3)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 129.0 t/s decode and 786 t/s prefill on an RTX 3090. Setup is sglang-exl3 v0.7.0-ampere with exl3-3bpw weights, MTP speculation, vision, and a 208k context window. Protocol acceptance records prose 96.2 and code 141.1 tok/s per stream, cold prefill 1,523 tok/s at 32k and 710 at 209k, and a 4096² image TTFT of 22.5 s. A companion Qwen3.6-35B-A3B recipe at 256k reached 309.6 t/s decode and 1,907 t/s prefill.

generation:

24.5 tokens/s

prompt processing (prefill):

300.2 tokens/s

quant:

IQ2_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at 24.48 t/s generation and 300.24 t/s prompt processing on 4x RTX 3090 (WRX80 platform, Threadripper PRO 3975WX, 251GB RAM). Setup is llama.cpp (club-3090 build) with UD-IQ2_XXS GGUF quant, 204800 token context, MoE cache enabled, speculative decoding on (draft acceptance 0.60833, mean len 4.04), 4 GPUs with PCIe-only interconnect (no NVLink, P2P unreachable). Model is 284.33B params, 84.62 GiB file at 2.56 BPW; GPUs are power-capped to 250W. Figures are from a single request (n_tokens 2797 prompt, 96 tokens generated).

generation:

6-7 tokens/s

quant:

4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4.1 Flash at 6-7 tok/s on a single RTX 3090 with 128GBx3 DDR4-2933 LRDIMM and a 4-bit quant. Setup uses ik_llama.cpp with experts streamed from SSD and on-demand expert streaming to GPU; there is a 100-200 token warmup while experts populate. User notes the machine is bottlenecked at 40GB/sec RAM throughput with three sticks and expects 20-30 tok/s with a fourth RAM chip. GLM 5.3 Flash support is also included but was slow.

generation:

1.8 tokens/s

prompt processing (prefill):

29.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at ~1.78 t/s generation and ~29.8 t/s prompt processing on dual RTX 3090s with 64GB DDR4 and 78GB swap. Setup is llama.cpp with UD-Q8_K_XL GGUF weights, q8_0 KV cache, 65536-token context, 11 GPU layers and tensor split 1,1. The user notes the generation figure used only GPU and RAM with no swap, and that prefill is fast because the GPUs handle it.