Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks which inference engine is fastest for a full quant on 96GB of VRAM, with 4x RTX 3090 and 256GB of 8-channel DDR4.
They report getting at most 6 t/s on short prompts with llama.cpp CUDA.
No model, quant, or context length is named.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8 27B at ~35 tok/s on an RTX 3090 with 64K context and Q5 quant, used as an always-on personal assistant via Vercel's eve framework.
Setup is 64K context, Q5 quant, xhigh reasoning, single RTX 3090.
User notes quality is great but speed feels slow for live chat; tried Gemma MoE at 120 tok/s but found it unreliable. Plans to test Q4 and lower reasoning, and recently learned reasoning can be disabled for compaction.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 2466 t/s prompt processing and 167 t/s generation on an RTX 3090 plus RTX 5070 Ti with the Strata engine and IQ3_S quant.
Setup is Strata with IQ3_S; the user also added UD-Q4_K_XL support, which reached 2341 t/s prompt and 126 t/s generation.
The user doubled Strata throughput over a week of profiling and reached 5x llama.cpp on UD-Q4_K_XL, where the starting point was 6 t/s. Earlier steps were 21 t/s, 27 t/s and 51 t/s on IQ3_XXS.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-uselong-context
User reports Qwen3.8 Flash-Next running on an RTX 3090 24GB with 128GB system RAM, reaching about 1650 t/s prompt processing and 38 to 61 t/s generation depending on context under Strata.
Setup is Strata with Unsloth UD-Q3_K_XL quant, fp16 KV cache, 256k context, speculative decoding with 4 draft tokens, and an expert cache of 6517 slots (~14GB). The same model on llama.cpp master gave up to 700 t/s PP and 23 t/s TG.
Generation is a range: 38 t/s at 182k context and about 61 t/s at short context. The user notes the quant Strata recommends by default was faster but produced minor errors and lower quality, so Unsloth's was chosen for accuracy.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 71.00 tok/s decode on one RTX 3090 24 GB.
Setup is NInfer v0.6.1 with INT8 KV cache, ReplaySSM and MTP3 speculative decoding, CUDA Graphs, 8,192-token context, 19.6 GB VRAM used.
Decode by cohort: C2 90.66, C4 100.28, C8 165.33 aggregate tok/s. Prefill with 4,362 fresh input tokens per request: C1 861.51 tok/s (TTFT 4,893 ms), C8 844.10 tok/s. MTP3 acceptance 61.13%, C1 TTFT 149 ms.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports Muse Glimmer 30B at 29.96 t/s median completion throughput on an RTX 3090.
Setup is llama.cpp (build 10437, CUDA) with GGUF Q4_K_M at 8,192 tokens of context, one concurrent request and no speculative decoding. The model used about 18.0 GB of VRAM on the 24 GB card, which ran at a fixed 280 W power limit.
Median latency was 6.94 seconds per question; prompt processing speed was not reported separately.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 36.4 tok/s on an RTX 3090 24GB, with throughput dropping to 20.9 tok/s at 120K context.
Setup is llama.cpp with Q4_K_M and q8_0 KV cache, all layers on GPU, 131,072-token context, flash attention, one generation slot.
The 36.4 tok/s figure is for a 2,073-token input with thinking disabled; cached continuations reuse almost the entire prefix and are not representative of fresh prompts. Peak GPU memory was 22,162 MiB. Three of four bounded coding tasks passed their checks, with one failing after spending its full 8,192-token output allowance on reasoning.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 101.7 tok/s on short prompts on an RTX 3090 24GB.
Setup is llama.cpp with Q4_K_XL, q8_0 KV cache, 65536 context, flash attention on, and all layers offloaded to the GPU.
The same model on Ollama reached 98.9 tok/s at 32k context and 23.4 GB VRAM. Qwen3.5-35B-A3B on llama.cpp reached 142.2 tok/s at 65k context and 21.7 GB VRAM. The author corrected an earlier comparison: llama.cpp was 1.4x faster than Ollama, not 2.4x, because Ollama had overflowed to the CPU in the first test. No speculative decoding; only generation speed was measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User releases ATX-Swift-1.5-Qwen3.8-27B-Uncensored-MTP, a GGUF quantization suite for Qwen3.8 27B, and benchmarks it on a single RTX 3090 24GB.
Setup is llamAmpere with i1-Q5_K_M weights, turbo5/turbo4 KV cache, 131,072 context, and MTP speculative decoding with the draft head at Q8_0. The full model and KV cache fit in 23.4 GB of VRAM with -ngl 99.
At 80k-98k active tokens the custom quant sustains 50.05 to 53.72 t/s with burst peaks of 58.10 to 63.05 t/s, versus 44.94 to 45.79 t/s for the standard Q5 quant. Draft acceptance rises from 63.7-65.1% to 73.1-81.7%, with mean draft length 3.31 to 4.10 tokens per round.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 127 tok/s single-stream decode on one RTX 3090 at 250 W.
Setup is vLLM 0.30.0 with 38 custom patches, int8 tensor-core GEMMs, speculative decoding with MTP or DFlash2 drafting at 7 drafts per pass, and KVarN 4/2-bit KV cache at 150k-262k context.
Decode reaches 381 tok/s at 25k context when the answer quotes the prompt. At 64 concurrent requests the aggregate is about 1,035 tok/s decode and 948 end to end. Prefill is about 1,440 tok/s, or 1,850-1,940 tok/s depending on int8 activation settings. The card is shared with training jobs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8-27B at 113.7 to 114.8 t/s decode and ~1,785 t/s prefill on 2x RTX 3090.
Setup is vLLM v0.29.0 with INT8 W8A16 compressed-tensors weights, fp8_e4m3 KV cache, 262,144 context, tensor parallel 2, DFlash2 W4A16 drafter with 7 speculative tokens, and a 64 GiB CPU offload tier.
User notes NVLink gives +12% decode and +18% prefill over PCIe P2P off, aggregate decode is 302.5 t/s at 4 concurrent, and a W8A8 build prefilled at 2,377 t/s at 46K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Q4 at 50-70 tok/s decode on a borrowed RTX 3090, with the RTX 5080 running ComfyUI at the same time.
The model was loaded entirely on the 3090. The user is deciding between a used RTX 3090, RTX 5090, Mac Studio M5 Ultra 96GB or 256GB, and DGX Spark for a two-user home setup with 1-4 concurrent requests.
Other figures in the post are gathered from benchmarks elsewhere, not measured by the user: RTX 3090 ~40 t/s (~66 with MTP), RTX 5090 ~60-75 t/s, M5 Ultra ~46-48 t/s, DGX Spark ~11-12 t/s (~24 with MTP). The user notes Flash-Next is 125B-A6B plus a ~48GB n-gram table that can live on SSD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen 3.8 27B at 31-34 t/s generation and around 1,080 t/s prompt processing on a single RTX 3090 24 GB.
Setup is llama.cpp with CUDA, UD-Q4_K_XL quant, 64K context, turbo3 KV cache, 2 parallel slots.
User compares four agent configurations: Qwen alone (1/5 scenes right, 26 min, $0), Sonnet 5.5 planning with Qwen coding (2/5, 85 min, $0.67), Qwen planning with Sonnet 5.5 coding (4/5, 14 min, ~$2.77), and Sonnet 5.5 alone (4/5, 9 min, $1.83). A 3060 12 GB was tried first and gave 4-6 t/s with ~7 GB of weights spilling into system RAM. User notes Qwen is a better planner than coder, that thinking budget had to be capped, and that crash checks reported zero errors while some scenes were visibly wrong. One run per setup, described as a field report rather than a benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Swift 1.5 + HyperQwen at 104.0 tok/s decode on an RTX 3090 24GB.
Setup uses FP8 KV cache and 150k configured context, with speculative decoding via MTP and a Swift-specific 65,536-token draft vocabulary.
Swift 1.5 INT4 heads achieves roughly 37% lower average time per task than the HyperQwen fast quant, averaging 68.2 s and 5,669 output tokens per task versus 108.1 s and 8,985 tokens. The HyperQwen fast quant decodes at 112.1 tok/s, Swift 1.0 at 105.9 tok/s, and Swift 1.5 INT8 heads at 104.0 tok/s. Average task time covers about 630 tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.6 35B-A3B at 112.4 tok/s on an RTX 3090 24GB with 64K context.
Setup is Ollama 0.32.14 with a Modelfile alias baking in num_ctx 65536; at 64K, 96.6% of layers were on GPU with about 0.71 GiB spilled to CPU.
The same runner at 32K reached 139.6 tok/s, so 64K was about 20% slower. The user also compares agent task success rates: warning the model to fix wrong values gave 1/9 successful runs with 4 false completions, deterministic tool-layer correction gave 8/15 with 2 false completions, and the same tool-layer approach at 64K gave 5/6 with 0 false completions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 162.9 tok/s decode on an RTX 3090 with native DFlash2 speculative decoding in ExLlamaV3.
Setup is ExLlamaV3 (community fork, pin 355c6ee) with EXL3 4.00bpw target and DFlash2 draft, cq3 KV cache, at the full 262,144-token context.
Acceptance length 5.66; autoregressive decode 42.8 tok/s and MTP head 116.3 tok/s for comparison. At 150k-token depth: 25.3 tok/s decode, 594 tok/s prefill, peak 23.13 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Swift-1.5-Qwen3.8-27b at 67 t/s on an RTX 3090.
Setup is the IQ4_XS quant, compared against Unsloth's Q4_K_S.
User says the model solved a custom Directory Opus script in 6 minutes that Gemini Flash failed to fix in 40 minutes, and was impressed that a low-thinking IQ4_XS 27B beat a major cloud model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-contextvisioncoding
User reports Qwen3.8-Flash-Next (125B MoE, ~6B activated) served on four RTX 3090s at full 262,144-token context, with an 806,792-token FP8 KV pool.
Setup is vLLM 0.30.0 (v2.2.0 release, 58 commits over the public tag) with a W4A16 AutoRound checkpoint, TP2 x PP2 plus expert parallel, MTP K=3, and FP8 KV cache with a scale sidecar.
Cold prefill runs 5,123-5,410 t/s from 10K to 200K tokens; decode with thinking on is 153.5-186.7 t/s (median of three prompts per depth, one boot). The release is a reliability update at parity with v2, not faster: decode lands at 0.973-1.040 of v2 in every cell. On a rented 4x RTX 3090 without peer-to-peer, a cold 54,713-token prompt took 13.8 s (about 4,000 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-uselong-contextvision
User reports Qwen3.8-Flash-Next at 193.3 tok/s median on 4x RTX 3090 (96 GB VRAM, 31 GB host RAM).
Setup is vLLM with Intel AutoRound W4A16 experts (Marlin), BF16 attention, FP8 n-gram table, fp8_e5m2 KV cache, 262,144-token context, 6 seats, and albucino's INT4 MTP draft with 3 tokens and expert parallel.
Without the draft the same lane measured 55.8 tok/s; real prompts ran 109.5 tok/s prose, 108.4 chat, 145.8 code. Six parallel coding agents sustained 317 tok/s aggregate (570 peak burst, 53 tok/s per agent, TTFT 0.47 s); at gmu 0.95 six agents gave 252.6 tok/s aggregate and 48 tok/s per stream. A llama.cpp lane on the same box measured 96 to 102 tok/s on copy/edit tasks and 40 to 49 tok/s freeform, with ~311 tok/s prefill and two 262,144-token slots.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 127 tok/s single-stream on a single RTX 3090 at 250 W.
Setup is vLLM with a 150k token context and an OpenAI-compatible API, in single-user mode.
Aggregate decode is ~1,035 tok/s at 64 concurrent in batch mode, and 381 tok/s when the answer quotes its own prompt.