User reports post-training Qwen3.8 27B into Swift-Qwen3.8-27B, achieving 58% fewer thinking tokens and 1.95x speedup while maintaining accuracy.
Training used an 8xH100 box with LoRA SFT, On-Policy Distillation, and a token penalizer targeting overthinking patterns. Benchmarks ran at xhigh thinking effort in BF16.
GPQA-Diamond: 88.3% vs 88.4% base with 58% fewer tokens; LiveCodeBench v6: 81.6% vs 76.8%; Terminal-Bench 2.1: 65.8% vs 66.7%; MMLU-Pro: 85.0% vs 85.5%; C-Eval: 90.6% vs 90.0%; IFBench: 71.8% vs 73.5%; AIME 2026: 94.0% vs 98.7%; HMMT: 96.0% vs 99.3%; ERQA: 66.3% vs 67.5%. Token savings hold across efforts: xhigh 41%, medium 23%, low 26%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 56.48 t/s decode and 980.2 t/s prefill on an RTX 4080 16GB at 131,072 context with 102,400 active input tokens.
Setup is ExLlamaV3 1.5.0 with TabbyAPI, the turboderp SC_3.00bpw_H4_V4 EXL3 quant, a 6,5 KV cache, and MTP k=2 with a Q6 draft cache. VRAM peaked around 15.2GB of 16.4GB.
Without MTP decode was 33.68 t/s, so MTP gave about a 68% decode increase at roughly a 5% prefill cost. Q4 draft cache was about 9% slower than Q6, and dynamic drafting was slower than fixed k=2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contexttool-use
User reports Qwen3.8 Flash-Next at ~120-126 t/s on 3x RTX 3090 (72GB VRAM) with MTP speculative decoding.
Setup is TabbyAPI with ExLlamaV3 1.5.0, 3.05bpw exl3 quant, 262,144 context, and MTP draft mode. Without MTP the model runs at ~90 t/s; MTP adds ~35% with ~40% acceptance. First request after boot reads ~58 t/s due to warmup.
Needle-in-haystack at ~97k tokens found with ~31s prefill; 4 concurrent requests complete in ~4s each. Tool calling works and reasoning splits into reasoning_content.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next-REAP-320 at 18.3 t/s generation and 25.4 t/s prefill on an RTX 5060 Ti 16GB with 32GB system RAM.
Setup is llama.cpp with a Q4 GGUF, 64k context, q4_0 KV cache, --n-cpu-moe 34, --ngl 48, and lazy mmap for the 29.48 GB n-gram embedding.
User notes the model is smarter than Qwen3.8 27B but keeps the 27B as daily driver due to 5x faster prompt processing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 12 t/s on an Intel Arc A770 using pure K-quants.
Setup uses the SYCL backend, where I-quant mixes like Unsloth's UD-Q3_K_XL run at 7 t/s while pure K-quants like Bartowski's Q3_K_S run at 12 t/s. The user notes SYCL is bottlenecked by inefficient I-quant handling.
The user is looking for the smallest K-quant-only Qwen3.8 27B at q3, with Bartowski's Q3_K_S at 12.7 GB as the current top contender.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next at ~100 t/s generation on 2x Radeon AI PRO R9700 with 128GB system RAM.
Setup uses the R9V engine with an IQ4_XS GGUF and n-gram SSD streaming, which the user says fixed crashes after ~12 hours of torture testing.
Q4_K_XL is now supported at 50 t/s generation but the user notes it still needs fine tuning.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8 27B at 30-40 t/s decode and 600-700 t/s prefill on 3x V100 16GB.
Setup uses Q8 quantization with speculative decoding, MTP, and prefix caching, tuned by an automated agent. On heavy agentic work at 256k context, decode dropped to around 20 t/s.
Qwen3.8 Flash Next at Q4 ran at 20 t/s decode and 90 t/s prefill. The build cost around $1500 and uses a 3D printed cooling block; GPU temperatures stay under 55C.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at 40 t/s on a dual-GPU PC with an RTX 5060 Ti 16GB and an RTX 3060 Ti 8GB.
Setup uses a Q3 quant and reaches around 160k context window.
User asks what the next meaningful step would be for local agentic coding, considering replacing the 3060 Ti with a second 5060 Ti or moving to twin DGX Sparks running a Flash model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8 27B at an average 100 t/s on an RTX 3090, compared to 20 t/s on a MacBook M5 Pro 48GB.
Setup is vLLM with a Q4 model on Linux, served over an OpenAI endpoint. The Mac run used llama.cpp with MTP and an Unsloth Q6 quant; Q4 on Mac made little speed difference.
User notes tool calls fail more often but the model self-corrects, and considers the setup good enough to replace a Claude subscription.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-usemathlong-context
User reports Qwen3.8 27B at 38.39 t/s generation and 871.93 t/s prefill on an RTX 3090 24GB, averaged over 3 hours of constant running.
Setup is vLLM 0.27.1 with the Club 3090 patch, INT4 AutoRound weights, FP8 E4M3 KV cache, 147,456 token context, single sequence, and AOT compilation on WSL2.
At 10K prompt the run reached 42.3 t/s generation and 1000.26 t/s prefill; at 90K prompt it reached 34 t/s generation and 743.59 t/s prefill. The user notes llama.cpp with Q5 UD GGUF at 81,920 context plus MTP gave 25-30 t/s, and that 154K context was the practical ceiling.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 20 t/s sustained on an M5 Pro 64GB, with about 30 t/s in the first 2-4k tokens.
User finds it unbearably slow, possibly because the model thinks so much.
User also tried antirez's DS4 with DeepSeek V4 Flash and gets around 8 t/s sustained, which they call slow and basically unusable.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 52 t/s decode and 1300 t/s prefill on Strix Halo.
Setup is forked llama.cpp builds and halogen-flash-server, described as roughly double decode and 5-6x prefill performance over prior Strix Halo results.
User offers a community reflection on local LLM optimization rather than a detailed benchmark report.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at roughly 50 t/s on an RTX 5070 Ti 16GB with 32GB of DDR5 system RAM, at about 26,000 tokens of context.
Setup is llama.cpp with the unsloth UD-IQ4_XS GGUF, a q8_0 KV cache, flash attention on, and an MTP draft model (mtp-Qwen3.8-27B-Q4_0.gguf) for speculative decoding.
User is happy with the speed but feels the context window is too small and asks for ways to get a larger one.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 12 t/s on an RTX 5080 16GB at 49,152 context.
Setup is llama.cpp with an NVFP4 GGUF, q4_0 K cache and iq4_nl V cache, flash attention on, 512 batch and ubatch, 16 threads.
User is asking for pi.dev plugin suggestions for context, compaction and memory management, and is disappointed that compaction settings are kept separate from model settings.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next reaching 1.2k t/s prefill on Strix Halo 128GB.
Setup is a custom llama.cpp branch with a custom HIP runtime, built to match the closed-source Halogen server's prefill numbers; the community fork had only reached 400 t/s.
The user plans to clean up the work and submit PRs to mainline and the community fork, noting the approach may also help the GLM 5.3 Flash architecture.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 7.19 t/s on an RTX 4070 Ti 12GB with 128GB DDR4 RAM.
Setup is the Nebula C/CUDA engine with native MTP speculative decoding, GPU expert caching (27 of 512 experts per layer resident in VRAM), and CPU MoE execution on an Intel i9-9940X using 14 threads, with a 24,576-token context capacity.
A limited-tolerance acceptance mode reaches 7.54 t/s. Time to first token is 25.11 seconds for a 512-token input and 113.94 seconds for 2,048 tokens. Prefill is noted as slow for longer prompts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-usevisionmultilingual
User reports Qwen3.8-Flash-Next (180B MoE, 7.31B active) at 43.9 t/s on a single DGX Spark, roughly four times the 11.2 t/s of Qwen3.8-27B dense on the same machine.
Setup is vLLM with NVFP4 weights, FP8 KV cache, MTP depth 3, greedy sampling, thinking mode off, tensor parallel size 1, and 320 tokens written per test. The 123.53 GiB checkpoint fits in 121.7 GiB unified memory because the 51.2B-parameter PLE n-gram table (47.68 GiB) is read from NVMe per token while the 120.8B experts (56.25 GiB at NVFP4) stay resident.
Tool calls, thinking mode, and image input work on the live server; video input was accepted but the test clip was faulty so that result is inconclusive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8 Flash-Next 177B-A6B at 38 t/s single stream on 2x RTX 3090 with an EPYC 7551 and 128 GB DDR4-2133.
Setup is llama.cpp with an IQ4_XS GGUF; experts live in system RAM with hot ones cached in VRAM.
Two parallel requests drop to about 4 t/s each. User asks whether a third RTX 3090 or a CPU upgrade would help more, and whether faster RAM matters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at about 60 t/s on a 16 GB RX 9070 XT at 128k context.
Setup uses the ISTA-DASLab GSQ-RCO IQ3_XXS GGUF with a HermiHg DFlash2 Q2_K_S draft model for speculative decoding, and a K 8_0 / V 4_0 KV cache.
User says this beats the built-in MTP, which multiplies VRAM use, and calls it one of the best single-thread choices on a 16 GB GPU. Speculative decoding is more sensitive to GTT overflow, so disabling it is faster when exceeding VRAM on a DDR5 PCIe 5 system.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 73.6 t/s on an RTX 5090 with 32 GB VRAM and 192 GB system RAM.
Setup uses FreeToken with a 123 GB model load; the user notes running out of context after a few prompts.
The user compares the speed favorably to a 27B model and calls the experience impressive.