Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 48.0 t/s (median 52.4) on 2x RTX 3090 with tensor parallel 2.
Setup is a fork of FreeToken with the NVFP4 model converted to FTW, 512 experts offloaded, max prefill length 8192, greedy decoding and MTP off.
With MTP on, greedy dropped to 21.5 t/s (median 22.4, accept rate ~56%), so MTP is a net loss on this config; under matched eager conditions it was ~1.5x plain (17.2 vs 11.4).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 27B running a local agent loop for about 21 days on a single RTX 3090, with prefill around 250 t/s.
Setup is vLLM serving the Q4 model with a Q8 KV cache at 200k context, using a deepseek harness and a written rulebook; llama.cpp reaches about 700 prefill t/s on the same card.
The run used 180 subagents, about 230M tokens in and out, 1.7B cache-read tokens, and 699 compactions totaling about 83 hours, roughly 17% of calendar time, with a typical compact taking about 7 minutes on a 160k+ token prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 80 t/s generation and 1500 t/s prefill on 3x RTX 3090 with 128GB DDR4.
Setup is ExLlamaV3 with an exl3 3bpw quant at 262,144 context, with vision and speculative decoding enabled.
A single RTX 5090 with 128GB DDR4 reached 29 t/s generation at the same 1500 t/s prefill and 262k context. User plans to test 4bpw later and says ExLlamaV3 is replacing vLLM and llama.cpp for this model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8 Flash Next at 110+ t/s generation and over 1,500 t/s prompt processing on 3x RTX 3090s.
Setup is ExLlamaV3 with a 3.05 bpw EXL3 quant at full context and FP16 KV cache, with only the engram table offloaded to RAM. All three cards run at 250-265W and one is on a TB4 eGPU.
User also ran Qwen3.8 27B (UD-Q8_K_L) on 2x 3090s in TP and a UD-Q4_K_XL of Qwen3.8 Flash Next that partially offloaded to RAM, reaching about 50 t/s max output, 38 t/s average, and prompt processing in the low hundreds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Kimi K3 at 1.05 t/s generation and 5.66 t/s prompt processing on an RTX 3090 with 2 TB of Intel Optane PMem 200 in Memory Mode.
Setup uses a Supermicro X12SPA-TF with Intel Xeon 8370C, 256 GB DDR4-3200, and the UD-Q4_K_XL quant at 16k context.
App Direct mode gave lower throughput at 0.62 t/s generation and 4.9 t/s prompt processing. The user also benchmarked Kimi K2.5, GLM-5.3-753B, Hermes-4-405B, DeepSeek-V4-Flash, Nemotron-3-Super-120B-A12B, gpt-oss-120b, and Ornith-1.5-35B-A3B on the same system.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next Q4 at 7-8 t/s generation and 110 t/s prompt processing on dual RTX 3060 cards.
No engine, context length, or KV cache quant is stated.
User asks whether a used RTX 4070 or RTX 5060 Ti 16GB paired with a 3060 could match RTX 3090 generation speed, and says the current 7-8 t/s is not enough.
User reports Qwen3.8 27B Swift IQ4_XS on an RTX 3090 at 200k context, with 8/TQ3 KV cache.
Setup is LlamAmpere v0.3.1 with custom kernels for EXL and ternary quants plus MTP heads. The release adds support for Turboderp's EXL format and Bonsai ternary 1.75/2bpw quants.
MTP gave about 20% speed up on 1.75bpw and 50%+ on 2bpw in initial 3090 testing. User also tested Qwen Swift on Livecode with xhigh reasoning, performing within 1.5% of original Qwen BF16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visioncodinglong-context
User reports Qwen3.8-27B (text + vision) as an 11.8 GB GGUF at 3.47 bpw, running at 42.3 tok/s single stream on an RTX 3090, matching Q4_K_M on the same card.
Setup is a patched llama.cpp build (13 files, 471 insertions on llama.cpp 6a1a922d) with the Opti-27B GGUF and mmproj, 16,384 context, full GPU offload. Prefill at 7k tokens is 1,127 tok/s; three concurrent streams give about 24 tok/s each.
Wikitext-2 perplexity is 6.487 versus 6.456 for FP16 and 6.457 for Q4_K_M at 16.8 GB. Four 16k conversations with vision fit in 17 GB; one 65k conversation with vision fits in 15 GB. MMLU-Pro is 53.9 +/- 0.5.
User runs SWEBench locally with llama.cpp and an Unsloth Qwen3.8 27B IQ4_NL GGUF on an RTX 3090, at 229,376 context with a q8_0 KV cache.
Setup uses --jinja, -ngl 999, flash attention, mmap load mode, and a bf16 mmproj; speculative decoding is disabled.
The 500-test run took 8 days and produced 9 LimitExceeded and 103 TimeoutExpired errors, leaving 388 completed tests with 339 resolved and 48 unresolved, an 87.5969% resolve rate. User asks how to reduce errors, improve success rate, and speed up the run without sacrificing context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 100 t/s on a 4x RTX 3090 machine with 96GB VRAM and 384GB DDR4 RAM.
Setup uses DFlash2 speculative decoding with BF16 weights.
User also tested Qwen3.8 Flash-Next at 4-bit, getting about 70 t/s, and found it produced visual glitches and HTML issues compared to 27B. Tried Intel AutoRound W4A16, generic W4A16, and Unsloth UD-Q4_K_XL quants for Flash-Next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8 27B at ~70 t/s decode on an RTX 3090 plus RTX 5070 Ti, with a ~400K shared context pool.
Setup is llama.cpp with the IQ4_XS GGUF, MTP and tensor parallelism across 3 cards at np 4, allowing up to 4 concurrent agents. A Q6_K_M quant of the same model also runs on the 3090 and 5070.
The user asks whether other engines handle mixed Nvidia GPUs better, noting vLLM, SGLang, and Freetoken did not work well for this rig.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next at ~80 t/s on 3x RTX 3090 at 1M context, dropping to ~60 t/s once QSA reaches its 2048-token budget.
Setup is vLLM with most of the KV cache offloaded to system RAM, no KV cache quantization. Prefill at 248k reaches 3,701 t/s, and throughput is ~150 t/s at 4 concurrent requests.
Decode speed stays flat as total context grows because only 12 of 48 layers hold a KV cache and QSA bounds the per-step read to 48 MiB per token.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~40 t/s decode on dual RTX 3090 (sm_86) with the IQ4_XS GGUF, stable from 1k to 126k context.
Setup is a vLLM 0.29.0 GGUF-capable fork with patches and env toggles; 32k fresh prefill measured at ~1104 t/s.
Q4_K_XL is kept experimental: ~32.3 t/s decode at 1k, ~854 t/s prefill at 8k, and unstable at long context. An older ~6665 t/s prefill figure was found non-reproducible.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at 90+ t/s on an RTX 3090 through 100K tokens, with context up to 240K.
Setup is a custom llama.cpp fork (llamAmpere) optimized for Ampere, using the IQ4_XS-M GGUF quant at temperature 1.
The user claims 80% faster than the nearest competitor at 200K context and says it runs faster than API speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contexttool-use
User reports Qwen3.8 Flash-Next at ~120-126 t/s on 3x RTX 3090 (72GB VRAM) with MTP speculative decoding.
Setup is TabbyAPI with ExLlamaV3 1.5.0, 3.05bpw exl3 quant, 262,144 context, and MTP draft mode. Without MTP the model runs at ~90 t/s; MTP adds ~35% with ~40% acceptance. First request after boot reads ~58 t/s due to warmup.
Needle-in-haystack at ~97k tokens found with ~31s prefill; 4 concurrent requests complete in ~4s each. Tool calling works and reasoning splits into reasoning_content.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8 27B at an average 100 t/s on an RTX 3090, compared to 20 t/s on a MacBook M5 Pro 48GB.
Setup is vLLM with a Q4 model on Linux, served over an OpenAI endpoint. The Mac run used llama.cpp with MTP and an Unsloth Q6 quant; Q4 on Mac made little speed difference.
User notes tool calls fail more often but the model self-corrects, and considers the setup good enough to replace a Claude subscription.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-usemathlong-context
User reports Qwen3.8 27B at 38.39 t/s generation and 871.93 t/s prefill on an RTX 3090 24GB, averaged over 3 hours of constant running.
Setup is vLLM 0.27.1 with the Club 3090 patch, INT4 AutoRound weights, FP8 E4M3 KV cache, 147,456 token context, single sequence, and AOT compilation on WSL2.
At 10K prompt the run reached 42.3 t/s generation and 1000.26 t/s prefill; at 90K prompt it reached 34 t/s generation and 743.59 t/s prefill. The user notes llama.cpp with Q5 UD GGUF at 81,920 context plus MTP gave 25-30 t/s, and that 154K context was the practical ceiling.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports GLM-5.3 Flash at around 28 t/s across four RTX 3090s at a quant of 3.
Setup is four RTX 3090 24 GB cards on a Threadripper PRO 3975WX with 256 GB RAM, PCIe only, no NVLink.
The user is planning a Qwen 3.8 Flash Next or Qwen 3.8 27B setup and asks for advice on model and hardware configuration.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8 Flash-Next 177B-A6B at 38 t/s single stream on 2x RTX 3090 with an EPYC 7551 and 128 GB DDR4-2133.
Setup is llama.cpp with an IQ4_XS GGUF; experts live in system RAM with hot ones cached in VRAM.
Two parallel requests drop to about 4 t/s each. User asks whether a third RTX 3090 or a CPU upgrade would help more, and whether faster RAM matters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 70 t/s sustained on a 2x RTX 3090 build.
Setup uses the Q4 quant from the syv ais repo and config, with 64GB DDR4 4000MHz system RAM.
The figure is described as sustained throughput; the user offers more details on request.