Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-UD-Q3_K_XL.gguf at up to 75 t/s tg and up to 1700 t/s pp on an RTX 5070 Ti 16GB, fully offloaded, with vision on CPU.
Setup is beellama.cpp with kvarn performance enhancements, MTP n-max=2, KV=kvarn3/kvarn3, MTP draft KV=kvarn2/kvarn2, 96256 context and 1024 tail tokens.
The user notes Q3_K_XL scores almost 100% on a needle test with kvarn3, while Q3_XXS does poorly at 84% with F16 KV and ~80% with Q3 KV, and that kvarn is better with KLD. The user advises not fearing Q3 quants or KV quantisation, and concludes that for 16GB VRAM Q3_K_XL with kvarn3 works well.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 60 t/s decode and 1500 t/s prefill with Qwen3.8-27B-IQ4-XS-MTP across an RTX 5070 Ti and an RTX 5060 Ti.
Setup uses tensor parallelism.
User is considering adding a 3090 FE for more VRAM but is concerned about mixing Ampere with Blackwell.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User asks for config optimization advice, reporting prompt processing at ~189 t/s and generation at ~15.8 t/s.
The setup splits a MoE model with 42 expert layers on CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B on dual RTX 5070 Ti 16GB cards.
Setup is llama.cpp with the Q4_K_XL quant, MTP3 speculative decoding, and 220k context.
The user also tests vLLM and NInfer, and praises llama.cpp for its simplicity and performance.
User reports Qwen3.8 27B UD Q4_K_XL fits in 24 GB of VRAM at 100k context with a Q8 KV cache.
User runs Qwen3.8-Flash-Next Q4 on dual RTX 3090s and is considering larger models such as MiniMax-M3 for agentic coding.
User calls MiniMax-M2.7 a workhorse but too slow for coding.
User benchmarks 21 Qwen3.8 27B variants on 16 GB of VRAM, sorted by mean KLD.
The best overall is bartowski/Qwen3.8-27B-IQ4_XS. The best uncensored is huihui-ai/Huihui-Qwen3.8-27B-abliterated-UD-IQ4_XS.
Also mentioned are jpetrina/Qwen3.8-27B-IQ4_XS-pure and Bucoid/Qwen3.8-27B-Uncensored-IQ4_XS_4BPW.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports 20 t/s average with MTP running a model in an FPS game built with it, on a 5900x CPU, 128 GB DDR4 RAM, an RTX 5090 and an RTX 4000 PRO with 32+24 GB VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next, 180B total with 6B active across 512 experts, generating 17.6 t/s on short prompts and 16.1 t/s at 12.5K context.
Non-expert weights run on a T4 using 4606 MiB, with the experts held in host RAM.
Prompt processing reaches 159.6 t/s on a cold 12.5K prompt. User reports solid refactor and coding performance, and finds it less verbose than Opus.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports decode improving from 25-29 to 37-41 t/s with the UD-Q4_K_XL quant, expert cache, and MTP.
Prefill runs 180-195 t/s, load time is 2 min, and cache hit rate is 90-92%.
MTP acceptance is 50-58% on reasoning and 94% on code emission. Host RAM is 73 GB pinned plus 28 GB PLE. The branch is available.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports MTP support merged in ik_llama.cpp, with decode at 90 t/s on code, up from 45 without.
Draft acceptance is 93-99% on code and 60-65% on prose.
The same test on an RTX Pro 6000 gives 85 to 113 t/s on code and 83 to 59 t/s on story, and on a 12 GB 4070 it gives 9.5 to 12.5 t/s on code.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a custom hybrid quantization that fits MTP and long context into 16 GB of VRAM, with VRAM usage of ~15.93 GB.
Setup uses an asymmetric KV cache mix of kvarn5 (K) and kvarn4 (V), with tail precision of 1024 tokens and speculative decoding with 2 draft tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8-Flash-Next at 30-50 tok/s single-stream decode on 2x NVIDIA DGX Spark linked by a direct ConnectX-7 cable.
Setup is vLLM with the RadixArk NVFP4 quant, TP=2 across both boxes, MTP speculative decoding with 3 tokens at about 40% acceptance, 512K context stretched from 262K with YaRN, and a 47.7 GiB n-gram embedding table mmap'd from NVMe instead of loaded into memory.
With about 9 concurrent agent sessions the engine reaches 181 tok/s aggregate, peaking at 195.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-next at 160 t/s prefill and 16 t/s decode.
Setup uses a kvarn5 KV cache quant, with experts offloaded to host RAM and n-grams on disk. MTP works but slows decode.
User also mentions alternative setups for 16 GB VRAM with kvarn4 and vision offloaded, and for 12 GB with ub 512.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 26t/s on an RTX 5070 Ti 16GB with 64GB of DDR5 6000MHz system RAM.
Setup is llama.cpp with the IQ3_XSS quant, using mmap to fit the model into 16G of VRAM plus 64G of RAM, on a 9800X3D CPU.
The user says this is faster than a non-MoE 30B model, which only reaches 10 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports dflash2 speeds up Qwen 3.8 27B up to 4x, with a median of 140.6 tok/s against a baseline of 47.4 tok/s.
Gains vary by task and are sometimes only 1.5x.
The user is from the atomic.chat team.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 NVFP4 on four Tesla V100s (2017) via a custom v100-skinny engine matching RTX 5090 decode throughput at ~220 tok/s.
Setup uses QPN kernels to translate FP4/FP8 to FP16 for Volta tensor cores.
Long-context tests show MTP depth k=3 better at ~65K context. The figure is for 4 GPUs versus 1, is not power efficient, and prefill is slower.
User compares PI Agent and Opencode with Qwen3.8 27B on an RTX 3090.
Setup is llama-server with the vision module offloaded to RAM.
User reports PI Agent uses fewer tokens, has no 32k output limit, runs faster, does not freeze, and compresses context later at 90k versus 67k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks a new NVFP4 quant of Qwen3.8 27B on an RTX 5090.
The quant includes a quantized MTP draft head.
User reports it is 50% faster than a Q4 quant and 4-7% faster than other NVFP4 quants.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports an unlocked CMP 170HX, A100 silicon with a firmware unlock, reaching 193 TFLOPS tensor throughput, up from 6.3 TFLOPS.
In llama.cpp, pp512 rises from 599.6 to 3468 tok/s.
Serving Qwen3.8-27B-FP8 under vLLM, one 170HX beats a 2x3090 tensor-parallel pair on prefill at 197W versus 454W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports 30-35 t/s from a local model, dropping to 21-22 t/s at long context.
The user is impressed with its coding performance, one-shotting tasks that Qwen3.6 35B failed.
The user notes occasional misunderstandings in conversation and on basic tasks, possibly due to a low quant.