Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8 27B at ~70 t/s decode on an RTX 3090 plus RTX 5070 Ti, with a ~400K shared context pool.
Setup is llama.cpp with the IQ4_XS GGUF, MTP and tensor parallelism across 3 cards at np 4, allowing up to 4 concurrent agents. A Q6_K_M quant of the same model also runs on the 3090 and 5070.
The user asks whether other engines handle mixed Nvidia GPUs better, noting vLLM, SGLang, and Freetoken did not work well for this rig.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at ~5.3 t/s on an RTX 5060 Ti 16GB with 16GB single-channel system RAM.
Setup is llama.cpp with IQ4 weights, 110k context, Q8 KV cache, and partial GPU/CPU offload; CPU and GPU each sit around 50% utilization.
User estimates Q8 weights plus 256k context would need 38-40GB total and drop to roughly 2 t/s, and asks whether quant level, context, or throughput should be prioritized for a local coding-agent worker.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 50 t/s generation and 1800 t/s prompt processing on an M5 Ultra at 8k context.
Figures come from the omlx website rather than a local run, with q4 weights and no MTP. The user notes the benchmarks' provenance is unclear but finds them reasonable.
User calls the results very promising.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at ~162.1 t/s generation and ~4.26k t/s prefill on an RTX 5090.
Setup is the Ninfer Windows Edition with a Q8 KV cache at 200k context.
Switching from LM Studio Q6 to Ninfer roughly doubled both decode and prompt speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next 8-bit on 2 RTX Pro 6000 Blackwell GPUs at 150 t/s decode and 10K t/s prefill.
Setup uses peer-to-peer communication, CUDA graphs, tensor parallelism, and both GPUs capped at 500W. The user also runs DeepSeek V4 Flash 0731 with 1M context and room for 4 concurrent requests.
User finds the system fast and plans to cancel subscriptions; asks about excluding thinking tokens from Qwen context and other models to try.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 15 t/s on an RTX 5070 12GB, with prompt processing at 100-120 t/s.
Setup is llama.cpp with a 3 bpw IQ3_XXS GGUF (76GB total, 47GB sharded into VRAM+RAM, rest on SSD) at up to 128K context.
Generation held 13.8 t/s at 1K, 14.6 t/s at 8K, 14.0 t/s at 20K, and 11.3 t/s at 90-100K context. User rates output quality as matching Unsloth Q6-Q8.
User reports Qwen3.8-27B RVN Heretic (ARA abliterated) at 140-190 t/s on code and ~100 t/s on prose on a single RTX 5090 at 159,744 context.
Setup is llama.cpp master with the Q6_K GGUF, q8_0 K/V cache, DFlash2 speculative decoding at n=4, vision projector on GPU, and Sharp chat template. Without speculation the same model runs at 62 t/s.
Cold prefill measured without the drafter: 32K in 11.7 s, 64K in 28.6 s, ~113K in 60 s. VRAM is 29,996 MiB after load and 30,332 MiB after the first image. Tool calls parse and images read correctly with no Xid.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next at ~80 t/s on 3x RTX 3090 at 1M context, dropping to ~60 t/s once QSA reaches its 2048-token budget.
Setup is vLLM with most of the KV cache offloaded to system RAM, no KV cache quantization. Prefill at 248k reaches 3,701 t/s, and throughput is ~150 t/s at 4 concurrent requests.
Decode speed stays flat as total context grows because only 12 of 48 layers hold a KV cache and QSA bounds the per-step read to 48 MiB per token.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticvisionlong-context
User reports Qwen3.8 27B peaking at 95.1 t/s generation and 1729.5 t/s prompt processing on an RTX 5070 Ti plus RTX 4070 Super, with means of 73.8 t/s and 845.1 t/s.
Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF and speculative decoding via draft-dflash plus ngram-mod; Dflash2 with ngram ran 20% faster than MTP with ngram. MTP acceptance was 0.48108 (674 accepted of 1401 generated).
User praises the model for coding and vision tasks but notes reasoning makes it slow and token-hungry, with up to 50% of context consumed by reasoning and poisoned tool calls looping at 100k+ context. Caching was finicky and default settings did not work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 90.8 t/s on a Radeon AI Pro R9700.
Setup is vLLM with a Q8 quant, single stream.
User says their vLLM numbers slightly beat the linked run, which wins on prefill by a lot, and recommends the llama-cpp-rdna-boosts repo for R9700 owners.
User reports ShapeLearn GGUFs for Qwen 3.8 27B, with the 3.84 bpw (GPU-5) quant reaching 99.63% of BF16's aggregate score across 8 benchmarks and the 3.23 bpw (GPU-4) quant reaching 98.72%.
Benchmarks were run on RTX Pro 6000 Blackwell, RTX 5090, RTX 4090, RTX 3090, RTX 4080 and RTX 5060 Ti. DFlash2 delivered 1.34-2.10x baseline throughput and MTP delivered 1.28-1.66x with temperature sampling.
User notes that Unsloth Dynamic V3's UD-IQ3_S had ~20% lower KLD than a similarly sized Lite model but scored 95.55% versus Lite's 97.33% of BF16's aggregate benchmark score.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~40 t/s decode on dual RTX 3090 (sm_86) with the IQ4_XS GGUF, stable from 1k to 126k context.
Setup is a vLLM 0.29.0 GGUF-capable fork with patches and env toggles; 32k fresh prefill measured at ~1104 t/s.
Q4_K_XL is kept experimental: ~32.3 t/s decode at 1k, ~854 t/s prefill at 8k, and unstable at long context. An older ~6665 t/s prefill figure was found non-reproducible.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at 90+ t/s on an RTX 3090 through 100K tokens, with context up to 240K.
Setup is a custom llama.cpp fork (llamAmpere) optimized for Ampere, using the IQ4_XS-M GGUF quant at temperature 1.
The user claims 80% faster than the nearest competitor at 200K context and says it runs faster than API speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingsummarizationcreative-writing
User reports Qwen3.8-Flash-Next (125B-A6B MoE) at 20.65 t/s generation on an RTX 4070 12GB with 64GB DDR5-5600 and Gen4 NVMe on Linux.
Setup is llama.cpp with AtomicChat's 4.27 bpw GGUF, n-gram SSD offloading, --fit-target 512, and an MTP variant (PR #28243 + compact shared-Q4_K_M head) with -ncmoe 45, yielding 77-96% acceptance. Prompt processing is 300-350 t/s.
Baseline was 6 t/s; master branch reached 19.35 t/s. User notes MTP gives limited gains at low VRAM because layers must be freed for the MTP head.
User reports post-training Qwen3.8 27B into Swift-Qwen3.8-27B, achieving 58% fewer thinking tokens and 1.95x speedup while maintaining accuracy.
Training used an 8xH100 box with LoRA SFT, On-Policy Distillation, and a token penalizer targeting overthinking patterns. Benchmarks ran at xhigh thinking effort in BF16.
GPQA-Diamond: 88.3% vs 88.4% base with 58% fewer tokens; LiveCodeBench v6: 81.6% vs 76.8%; Terminal-Bench 2.1: 65.8% vs 66.7%; MMLU-Pro: 85.0% vs 85.5%; C-Eval: 90.6% vs 90.0%; IFBench: 71.8% vs 73.5%; AIME 2026: 94.0% vs 98.7%; HMMT: 96.0% vs 99.3%; ERQA: 66.3% vs 67.5%. Token savings hold across efforts: xhigh 41%, medium 23%, low 26%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 56.48 t/s decode and 980.2 t/s prefill on an RTX 4080 16GB at 131,072 context with 102,400 active input tokens.
Setup is ExLlamaV3 1.5.0 with TabbyAPI, the turboderp SC_3.00bpw_H4_V4 EXL3 quant, a 6,5 KV cache, and MTP k=2 with a Q6 draft cache. VRAM peaked around 15.2GB of 16.4GB.
Without MTP decode was 33.68 t/s, so MTP gave about a 68% decode increase at roughly a 5% prefill cost. Q4 draft cache was about 9% slower than Q6, and dynamic drafting was slower than fixed k=2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contexttool-use
User reports Qwen3.8 Flash-Next at ~120-126 t/s on 3x RTX 3090 (72GB VRAM) with MTP speculative decoding.
Setup is TabbyAPI with ExLlamaV3 1.5.0, 3.05bpw exl3 quant, 262,144 context, and MTP draft mode. Without MTP the model runs at ~90 t/s; MTP adds ~35% with ~40% acceptance. First request after boot reads ~58 t/s due to warmup.
Needle-in-haystack at ~97k tokens found with ~31s prefill; 4 concurrent requests complete in ~4s each. Tool calling works and reasoning splits into reasoning_content.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next-REAP-320 at 18.3 t/s generation and 25.4 t/s prefill on an RTX 5060 Ti 16GB with 32GB system RAM.
Setup is llama.cpp with a Q4 GGUF, 64k context, q4_0 KV cache, --n-cpu-moe 34, --ngl 48, and lazy mmap for the 29.48 GB n-gram embedding.
User notes the model is smarter than Qwen3.8 27B but keeps the 27B as daily driver due to 5x faster prompt processing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 12 t/s on an Intel Arc A770 using pure K-quants.
Setup uses the SYCL backend, where I-quant mixes like Unsloth's UD-Q3_K_XL run at 7 t/s while pure K-quants like Bartowski's Q3_K_S run at 12 t/s. The user notes SYCL is bottlenecked by inefficient I-quant handling.
The user is looking for the smallest K-quant-only Qwen3.8 27B at q3, with Bartowski's Q3_K_S at 12.7 GB as the current top contender.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next at ~100 t/s generation on 2x Radeon AI PRO R9700 with 128GB system RAM.
Setup uses the R9V engine with an IQ4_XS GGUF and n-gram SSD streaming, which the user says fixed crashes after ~12 hours of torture testing.
Q4_K_XL is now supported at 50 t/s generation but the user notes it still needs fine tuning.