NVIDIA RTX 3090 reports

generation:

116.6 tokens/s

quant:

W4A16 (W4A16)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 180B W4A16 at 116.6 t/s decode on four RTX 3090s capped at 220 W each. Setup is vLLM built from a pinned PR head with five patches, MTP speculative decoding with k=4, 65,536 context, and 76,046 KV tokens. Decode holds 115.1 t/s at 32k and 72.75 t/s at 128k; prefill reads 3,187 t/s at 32k and 2,835 t/s at 128k. The c8 aggregate is 180.7-230.3 t/s on the bf16 profile. Peak short-prompt decode reaches 198.5 t/s on code. Spread is plus or minus 15% run to run.

generation:

120.0 tokens/s

quant:

Q3_K_M (GGUF)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 120 tok/s on an RTX 3090. Setup is llama.cpp with Unsloth's Q3_K_M GGUF quant and bf16 KV cache, using 23GB of the card's 24GB VRAM at 262144 tokens of context. The user says it is the first local model to saturate their personal task benchmark, running it for three days through OpenCode, though they note some tasks needed reinforced prompts and adapted tooling.

generation:

5.6 tokens/s

quant:

fp8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 5.58 tok/s on 2x RTX 3090 with the offload MoE backend, versus 0.67 tok/s with the auto-selected hybrid backend. Setup is FreeToken 0.1.2 with fp8 dense weights and fp4 experts, TP=2, moe-cache-auto resolving to 624 slots (5.7% residency). Decode measured over a streamed 60-token completion, first token excluded, after a warmup request. The auto-selected hybrid backend is 8.3x slower than offload. Raising memory-ratio to 0.95 grew the expert pool to 779 slots (7.1% residency) for only +6.6% throughput (5.95 tok/s). Doubling CPU threads from 23 to 44 yielded only +15%. The user notes TP=2 is a regression for offloaded MoE per issue #62, so absolute numbers are pessimistic.

generation:

33.0 tokens/s

quant:

MXFP4 (safetensors)

kv:

fp8_ds_mla

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4-Flash-0731 at 33 tok/s single-stream on 2x RTX 3090 in a used quad-Xeon R940 server with 768 GB DDR4. Setup is the Lvllmds4-x vLLM fork with the official MXFP4 safetensors checkpoint, fp8_ds_mla KV cache, 32K context, and DSpark speculative decoding with 5 draft tokens. Aggregate throughput is 53-68 tok/s at 4 concurrent requests and 47-63 tok/s at 8, with per-user rates of 13-17 and 6-8 tok/s. Cold prefill plateaus at 420-480 tok/s, warm prefill reaches 2,030-10,200 tok/s, and the same box on ik_llama.cpp hybrid does 12.2 tok/s single-stream. The user notes the bottleneck is CPU-side DRAM bandwidth, not the GPUs.

generation:

138.0 tokens/s

quant:

GPTQ-int4 (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~138 t/s single-request on an RTX 3090, up from ~124 t/s, using a custom vLLM 0.27.1 stack with DFlash2 speculative drafting. Setup is vLLM with GPTQ-int4 weights, fp8 KV cache, 64k context, and a W4A16 DFlash2 drafter; the 3090 is power limited to 250W. The same stack reaches 942 t/s at 64 concurrent requests. User notes DFlash2 is best for 1-4 concurrent users and MTP wins at 8+ concurrent; quality is unchanged with perplexity 8.09 and GSM8K 96.5%.

quant:

UD-Q8_K_XL (GGUF)

kv:

q8_0

flash attention:

on

User reports Qwen3.6-27B-MTP UD-Q8_K_XL running on 2x RTX 3090 with llama.cpp build b9455, using tensor-split 50,50 and MTP speculative decoding. Decode speeds range from 54 to 81 t/s across runs, with prefill between 823 and 1421 t/s at contexts up to 68K. The user notes vLLM previously achieved 70+ t/s but lacked good quants, and praises the code output quality with this setup.

generation:

20-50 tokens/s

quant:

Q6 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 at 20 to 50 tokens per second on dual RTX 3090 GPUs. Setup is llama.cpp with Q6 quantization and MTP speculative decoding; the cards are undervolted and limited to 65°C. The user compares Q4 to Q6 quality for a coding agent, calls the Q6 improvement outstanding, and says the local server now works similarly to paid APIs.

generation:

72.0 tokens/s

quant:

IQ4_KS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B on a single RTX 3090 at an enforced 370 W power cap, comparing ik_llama.cpp with IQ4_KS against mainline llama.cpp with Q4_K_M. ik_llama.cpp reaches ~72 t/s code decode and ~60 t/s narrative decode versus ~58 and ~50 t/s for mainline, an 18-20% gain, with quality tied at 100-103/150 on an 8-pack and ~0.5 GB lower VRAM. Setup uses q4_0 KV cache, MTP n=2, -np 1, and a 200K context configuration that fills to ~183K tokens. A two-stage ngram plus MTP speculator on the same engine lifts code decode to ~98 t/s at the same power cap, with narrative flat at ~59 t/s. The user notes the earlier reported tie was a measurement artifact from a stale mainline container answering on the shared port, and that both engines are strongly power-sensitive between 230 W and 370 W.

generation:

80-120 tokens/s

quant:

6bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-120 t/s on 6x RTX 3090. Setup is ExLlamaV3 with a 6bpw EXL3 quant from turboderp. The user describes the throughput as a range and praises the engine for CUDA, using it with the Hermes agent controlled from a phone via Matrix.

generation:

80.0 tokens/s

quant:

W4A16-FP8PLE (W4A16)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80 t/s in real world scenarios on dual RTX 3090s. Setup uses the W4A16-FP8PLE quant with context reduced to 220k and hot experts increased to 88 for max speed. User also reports 2k+ prefill.

quant:

Q8_0 (GGUF)

User reports running Qwen3.8-27B at 44-46 t/s at the start of context and 28-30 t/s as context fills, on three RTX 3090 GPUs. Setup is self-compiled llama.cpp with Q8_0 GGUF, 262144 context, tensor split 14,24,24, and ngram-mod plus draft-mtp speculative decoding. The 3090s are power-limited to 300W. The user notes the machine is a primary desktop, so 2-3 GB of VRAM is reserved for the OS and applications, leaving roughly 62GB usable and 10-12 GB spare at full context.

generation:

14-22 tokens/s

prompt processing (prefill):

130-200 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 130-200 t/s prefill and 14-22 t/s generation (about 15 t/s on average) on dual RTX 3090s. Setup is llama.cpp with Q4_K_XL GGUF and q8_0 KV cache at 131072 context, 48GB VRAM across two cards, 128GB DDR4 and an old 40-core Xeon in a Proxmox VM with PCIe passthrough and NUMA binding to one physical CPU. Prefill is thought to be CPU-limited; ngl, ncmoe and tensor split were manually adjusted to fit the model without crashing.

generation:

45-60 tokens/s

prompt processing (prefill):

1000-1500 tokens/s

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports running Qwen3.8-27B on a dedicated headless RTX 3090 for coding work, with 1000-1500 t/s prefill and 45-60 t/s generation at 180224 context with Q8_0 KV cache. Setup uses one RTX 3090; the user says it replaced cloud subscriptions for most real coding tasks. The user also ran Ornith-1.5-9B on a home server with 8GB VRAM and found it successful on simple Python refactoring tasks, but has not benchmarked it and asks for others' real coding experience.

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

User reports running DeepSeek V4 Flash 0731 with UD-Q4_K_XL quant on 2x RTX 3090 + 1x RTX 3060 (128GB RAM, ~60GB VRAM) using a custom llama.cpp fork. Generation speed is low to mid twenties t/s, prompt processing around 45 t/s with the moe-cache branch, and near 200 t/s with a separate IQ2_M prompt-processing model. Setup uses 167936 context, q8_0 KV cache, dflash speculative decoding, and MoE expert caching with mlock. User notes prompt processing remains a bottleneck and overall prompt handling only improves for prompts over 30K tokens.

quant:

Q4_K_M (GGUF)

User releases Mica v0.1 4B, a Qwen3.5-4B fine-tune with a rank-16 LoRA, as a decision model that returns calibrated probabilities for yes/no, choice, or score answers. It runs on an 8 GB GPU and was trained for under $30 of rented RTX 3090 time. Setup is llama.cpp with GGUF weights and a direct logits readout. The Q5_K_M file is 3.5 GB with no measurable accuracy loss against BF16 on the calibration set. On an RTX 3090 with one request at a time, median latency over 231 public JevBench items is 47 ms for Q4_K_M and 54 ms for BF16. On held-out English decisions Mica scores 67.0 versus 74.1 for Jev 1.13 and 55.0 for the Qwen3.5-4B base. On long inputs around 3.7k tokens Mica is slightly slower than JevK5.

generation:

40.0 tokens/s

prompt processing (prefill):

300.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 40 t/s generation and 300 t/s prompt processing on an RTX 3090 in a 4-GPU system totalling 72 GB VRAM. Setup is llama.cpp with an IQ4_XS GGUF. A second configuration with ExLlamaV3 at 3.05 bpw achieves 25 t/s generation and 1,200 t/s prompt processing. The user is considering an Epyc 7443 upgrade with 256 GB DDR4 for more PCIe lanes and CPU offloading, and asks whether the extra RAM and bandwidth would be worthwhile.

generation:

50.3 tokens/s

prompt processing (prefill):

1025.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 27B at 50.27 t/s narrative and 58.92 t/s code on a single RTX 3090 at 200K context. Setup is llama.cpp with MTP speculative decoding and the Unsloth Q4_K_M GGUF; prefill measured 1025 t/s. Soak testing gave a p50 of 54.5 t/s with 102 percent retention, and a verify-stress ladder showed 183K fill capacity with margin.

generation:

140.0 tokens/s

quant:

Q2_64 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B in PrismML's Q2_64 quantization running at ~140 tok/s in chat on an RTX 3090, using under 10 GB VRAM including KV cache and vision projector. Setup is a fork of PrismML's llama.cpp with a custom /v1/systemone endpoint for typed decisions, tested with 22,000 requests at massive parallelism; p50 latency ~171 ms including HTTP network latency. User also cites ~143 tok/s for chat completion on an RTX 5090, and benchmark results of 76% Soft Accuracy vs ~88% for Jev-1.13 on a 22,000-request typed-decision benchmark.

generation:

135.2 tokens/s

prompt processing (prefill):

1402.0 tokens/s

quant:

W4A16 (AutoRound)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 1,402 prompt tok/s and 135.2 output tok/s on 2x RTX 3090 with 128 GB system memory. Setup uses the Intel AutoRound W4A16 target with FP8 PLE/ngram tensors. Prefill and decode use different test shapes and did not come from the same request. A 262,016-token prompt with 128 output tokens prefilled at 1,275.6 tok/s and decoded at 54.5 tok/s. Warm decode over 4,096 output tokens ranged 127.1-134.0 tok/s. Maximum tested sequence was 262,144 tokens.

generation:

70.0 tokens/s

quant:

INT4

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at over 70 t/s single-stream decode on 2x RTX 3090, with close to 200 t/s decode at 3 to 4 concurrency and over 10k t/s prefill. Setup is stock vLLM v0.29 on Linux serving the RedHatAI INT4 quant with an FP8 KV cache, MTP speculative decoding with 3 tokens, tensor parallel size 2, and the full 262k context. The user notes the setup is standard and that the RedHat quant is overlooked for Ampere hardware.