Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports about 40 tokens per second running a Swift-1.5 Qwen3.8 27B IQ3_S GGUF on an RTX 5060 Ti 16GB.
Setup is llama.cpp with MTP speculative decoding at 81920 context size, on a Ryzen 7 9800X3D with 32GB DDR5-6000.
User asks whether this is the best they can do for general use, coding and agentic tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 58.53 tok/s on an RTX PRO 4000 Blackwell SFF, up from 35.04 tok/s with DFlash2 speculation after power and clock tuning.
Setup is NVFP4 weights at 8,000 input tokens with ECC off, best at a 120 W limit, VRAM +20% and core +180 MHz; the baseline was 70 W with zero offsets.
A second model, Qwen3.6 35B A3B in Q4_K_S with no drafter, went from 84.97 to 115.56 tok/s at 120 W, VRAM +33.3% and core offset 0. The user notes 120 W exceeds the 75 W PCIe slot budget on this connectorless card and calls it an experiment, not a recommended daily profile.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-uselong-context
User reports Qwen3.8 Flash-Next (Swift 1.5) at 144.6 tok/s decode and 3,444 tok/s prefill on a Mac Studio M5 Ultra 96GB, at a median context of 147.5K tokens.
Setup is Splash with IQ3_S GGUF weights and a 262,144-token context window, driven by the DeepSeek Harness agent framework for agentic coding.
With two concurrent requests combined decode is 200.4 tok/s (100.2 per request) and prefill 3,006 tok/s; with three it is 218.0 tok/s combined (72.7 per request) and 3,253 tok/s prefill, a result the user calls preliminary. The user is happy with the numbers and finds the model less lazy than Sol at troubleshooting.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B generation rising from 35.04 to 58.53 tok/s (+67%) on one RTX PRO 4000 Blackwell SFF after raising the power limit from 70 W to 120 W with VRAM +20% and core +180 MHz offsets.
Setup used a modified VBIOS loaded into memory via a patched loader and NVIDIA open kernel modules, with no EEPROM flash; headline comparisons used 8,000 input tokens with ECC off.
Only 1-3 requests per configuration across 158 total, and the 70 W baseline used the same modified VBIOS rather than factory firmware. Pushing VRAM from +20% to +33.3% slightly reduced 27B generation, 57.70 to 56.91 tok/s. Qwen3.6 35B A3B was almost flat between 106 and 120 W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 2076 tok/s prompt processing on a 145k-token prompt and 95-105 tok/s decode depending on task, with peaks of 145 tok/s, on a single RTX 4090 24 GB.
Setup is Strata 0.1.43 with an IQ3_S pack, int8 KV cache, 204800 context, resident experts and MTP speculation, on a Ryzen 9 3900X with 48 GiB DDR4-3200 under Windows 10.
A grid search over flags raised decode from 97 to 142 tok/s; the best arm was --kv-resident 20480 plus --adapt-async 1 and STRATA_ADAPT_LAG=2, giving 1526 and 2309 tok/s prompt processing at 10k and 20k. The user notes Windows does not pin memory with CUDA, so every host-to-GPU expert copy is CPU-driven.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodinglong-context
User benchmarks Qwen3.8 27B (Swift-1.5, AWQ-INT4) on dual RTX 3080 20GB cards in tensor parallel, reporting 2080 t/s prefill at 8192 prompt tokens with gpu-unlock plus P2P versus 967.1 t/s stock, a 115% gain.
Setup is vLLM 0.32.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, and both cards limited to 240W in PCIe 3.0 x16 slots.
Prefill gains range from 70% to 115% across prompts of 8192 to 261900 tokens. Decode is reported as inter-token latency step rates of about 15 to 16.6 steps/s, which the user says corresponds to roughly 45 tok/s at an MTP acceptance length of 2; decode changes are within noise except at 261k tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-27B at 66 t/s decode at 262,144-token context depth on one RTX 3090, with the full context fitting in 23.5 GB.
Setup is a custom engine with unsloth UD-Q4_K_XL weights, K cache at 6 bits, V at 5 bits, and the newest 2,048 positions kept at q8_0, saving 1.1 GB versus a full q8_0 cache at 120k context.
KV quality was scored by KL divergence against llama.cpp q8_0 output: 0.00160 at 64k versus 0.00163 for q8_0 and 0.00161 for fp16, +8% at 32k, and the same 230/282 on the author's question set. The author notes Qwen3.8 is a hybrid with only 16 of 64 layers attention, so full-attention models may differ.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B ASCII-Condensed UD-IQ4_XS at about 81 t/s code generation on an RX 9070 16 GB card with 32 GB system RAM, running on Windows.
Setup is achyudh's llama.cpp ROCm KV-streaming fork with q8_0 K / q4_0 V cache, 262144 context, --kv-stream with 300/1800 MiB resident KV, MTP n=4 plus n-gram speculation, and 6 threads.
Decode was 75 t/s at 9k and 95 t/s at 21k tokens, prefill about 1000+ t/s; these are single runs on short prompts, not filled 64k-262k contexts, and output quality was not checked. The user notes their numbers are below the fork README's ~100+ t/s at 16k-64k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports about 30 tok/s with the Swift-1.5 version of Qwen3.8 Flash-Next on a Tesla P100 16GB in a PCIe3 x16 slot, after switching from llama.cpp to Strata.
Setup is Strata with 64GB of ECC DDR4 at 2100 and a 131072-token context; the logs show expert cache hits of 63.5% to 77.8% with routed experts going over PCIe, so part of the model ran from system RAM.
Individual runs in the pasted logs range from 28.7 to 33.3 tok/s on prompts of roughly 78k tokens, which were read in 2 to 3 s. The user says it serves their Hermes agent and calls it impeccable for a local LLM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingcreative-writing
User benchmarks Qwen3.8-27B on two free Kaggle T4s (no P2P, 70W cap per card) with EXL3 4.0bpw under exllamav3, tensor parallel over PCIe, and a DFlash2 block drafter of 7 tokens.
Overall throughput is 55-57 tok/s with drafting versus ~18-19 without; HumanEval reaches 154 tok/s and prose 35-37 tok/s.
Prose acceptance is the limit at about 2.5 accepted tokens per step versus ~7 on code. On the same cards llama.cpp best was 16.5 tok/s and vLLM int4 with its own MTP drafter did 40.8 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks a llama.cpp PR that fuses FFN gate/up matmuls and GLU in MMQ, on a DGX Spark (GB10) and an RTX 5090.
Setup is llama.cpp with llama-bench, -ngl 999, flash attention on, 16384-token prompts, F16 KV cache, batch 2048, 5 repetitions.
On the DGX Spark, Qwen3.8-27B Q4_K_M prompt processing rises from 840.1 to 899.9 tok/s (+7.1%). On the RTX 5090 the same model and quant goes from 3688.4 to 3790.9 tok/s. The table also covers Qwen3.6-27B and Llama-3.1-8B across Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS; only prompt processing was measured (-n 0).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagenticlong-context
User reports Qwen3.8-Flash-Next at 49 t/s decode on an M5 24GB MacBook Air.
Setup is Splash 1.4.0 with a small fork, ISTA GSQ-RCO Q2_0 weights and a Flash-Next-DFlash2 draft model at 256K context. About 28% of expert slots (6,800 of 24,576, ~8.8 GB) stay resident and the rest are streamed from the SSD.
Cold prefill without prompt cache is 522 t/s on a 32,620-token prompt and 485 t/s on 63,201 tokens. Real agent use in pi averages ~45 t/s with no prompt-cache losses, and draft acceptance is ~.70 for code and ~.80 for tool calls.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 164.6 tok/s decode on an RTX 5090 32GB, with prefill of 6,000~6,500 tok/s.
Setup is Strata with an IQ3_S GGUF quant, on a Ryzen 9 9950X3D system with 64GB DDR5.
The user previously ran IQ3_XSS and switched to IQ3_S for more speed and quality. The prompt asked for a Minecraft-style Eiffel Tower, which the model built in detail.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target plus DFlash 2 draft) on a MacBook Pro M5 Max with 48 GB, using a Python code context and 128 generated tokens per test.
At a 4096-token prompt the single request reaches 93.8 t/s decode and 1207.0 t/s prefill; longer prompts give 189.4 t/s decode at 16382 tokens and 110.2 t/s at 65536 tokens.
Batched continuous batching reaches 413.7 t/s aggregate at 2x and 407.0 t/s at 4x, with peak memory between 19.1 GB and 22.2 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 125B at 20.0 tok/s decode on an RTX 4090 at the full 262k context window, with experts streamed from NVMe SSD.
Setup is ExLlamaV3 with a 3.05bpw_h5_ng5 quant and Q4 KV cache, 16 GB VRAM and only 8 GB system RAM allocated.
At 100k context the same card gives 20.8 tok/s decode and ~1,536 tok/s prefill; short 8k context gives 28-33 tok/s prose and 19-22 tok/s code with ~1,080 tok/s prefill. An RTX 3080 Laptop with 16 GB VRAM and 32 GB RAM reaches ~18-21 tok/s decode and ~650-700 tok/s prefill at 100k context, peaking at 15.5 GB VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-Flash-Next 180B-A6B at 46-53 tok/s generation and ~770 tok/s prompt processing on a Strix Halo mini PC with 128 GB unified memory.
Setup is llama.cpp and later the Strata engine, with UD-IQ4_XS GGUF weights in 3 shards at 94 GB, plus MTP speculative decoding and a vision projector.
The same box ran Qwen3.6-35B-A3B at 76.5 tok/s, and that model did 11.7 tok/s CPU-only on the user's Ryzen 7 6800U laptop. The 46-53 tok/s figure is a range, so no single generation rate is recorded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Qwen3.8-Flash-Next IQ3_S on an RTX 5080 16 GB with a Core Ultra 9 285K and 62 GB DDR5-6800, averaging 154.9 t/s generation and 5,475 t/s prompt processing in a context sweep of fresh 15K-207K prompts.
Setup is Basalt with IQ3_S weights, 262K context and int8 KV cache, using 15.7 GB VRAM and 54.1 GB system RAM.
An agent coding session over 103 requests reached 153.0 t/s average generation with a 262 t/s peak, and 5,011 t/s average prefill; 99.5% of prompt tokens came from cache with a 0.2 s median time to first token.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at an average 208.2 tok/s on a single RTX 5090 32GB.
Setup is SGLang in Docker with the gittensor NVFP4 build and the DSpark speculative drafter, served over an OpenAI-compatible API.
Five runs of 256 tokens with thinking on ranged from 148.3 tok/s (cold start) to 232.4 tok/s, averaging 208.2; the user says it sits in the 220s once warm and peaks above 230.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B averaging 95 t/s decode and 1500 t/s prefill on a single RX 7900 XTX using a hipfire fork.
No quant, context length or KV cache settings are stated.
The user calls the result a removal of AMD's governor and is enthusiastic about it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 353 t/s on four V100 16GB cards.
Setup is NInfer with NVFP4 weights, Q8 KV cache, 256k max context and MTP 4/0 speculation, tensor parallel across 4 GPUs.
The user previously ran 2 V100s at 181 t/s and notes the repo now supports tp4.