Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingvisionlong-context
User reports Qwen3.8-27B running on a single 24 GB RTX A5000 with llama.cpp b11491 (CUDA 13.4), using the IQ3_S-mtp quant (12.1 GB) with q4_0 KV cache at 256K context and MTP speculative decoding. Decode is 57-58 t/s at short context, falling to 27.5 t/s at 250K; prefill is ~900 t/s up to 32K and 478 t/s at 250K.
Setup uses --parallel 1 (requests serialized), -ngl all, -ub 2048, and the mmproj vision encoder offloaded to GPU. A second recipe with UD-Q4_K_XL (17.5 GB) is also provided, reaching ~38-40 t/s short-context decode and 950-973 t/s prompt eval.
MTP acceptance is 55-75% at temp 0 and 48-62% at temp 1.0. The user notes asymmetric KV quantization collapses prefill to ~34 t/s, and q8_0 KV above 128K is unstable on this architecture.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 54.1 t/s on a Beelink GTR9 Pro with Ryzen AI Max+ 395 and 128 GB unified memory.
Setup is Gufo with UD-Q4_K_XL and a shared Q8_0 MTP drafter at 8K context, one session, 80% draft acceptance.
Gufo loads crash the machine with a data fabric sync flood 4 of 7 times, at random points 9-40 GiB into load, never after a successful load; one load ran 46 hours. Vulkan llama.cpp runs the same model at ~45 t/s structured and ~32 t/s prose on the old config, and gpt-oss-120b at 54.9 t/s. Qwen3.8-27B ran at 12.3 t/s and Flash-Next without MTP at 26.8 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at about 2.5 tok/s on a OnePlus 13 phone with 24 GB RAM.
Setup is llama.cpp with Q4_K_M at 16K context, using around 21.2 GB of memory; the LLM and TTS run on the NPU.
User also runs Qwen3.5 9B at around 5 tok/s using about 16.1 GB RAM with always-on ASR and TTS. Heat is the main problem, so inference pauses when the phone gets too hot, and lower-power modes reduce TPS.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writing
User reports Qwen3.8 27B at 10.01 t/s on a Tesla P100 16GB.
Setup uses the Q3_K_M GGUF quant with a 2,180-token prompt that took 3 minutes 37 seconds.
The user asks whether this speed is normal and whether a 14B model would be faster, noting the P100 is old hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
math
User reports running a quant of Qwen3.8 27B on an RTX 3090, getting around 50 t/s. The model was benchmarked on a noisy data science prediction problem against cloud models Luna 5.6 and Astra, where it performed better than Luna and competitively with Astra. The user notes the local model took 5-15 times longer than the cloud models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodinglong-contexttool-use
User reports Qwen3.8-Flash-Next at 27.16 t/s on Strix Halo 128GB, from a ranked table of 33 models measured on the same hardware.
Setup is llama.cpp Vulkan/RADV with UD-IQ4_XS weights; the same model runs 50.69 t/s on the user's Halogen engine with 79.1% draft acceptance.
The table's other rows range from 266.00 t/s (Qwen3 0.6B) to 4.90 t/s (Llama 3.1 70B); the user notes dense models above 14B are below usable speed and that the intelligence column is unsourced opinion, not measured data.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next prefill at 1821 tok/s at 16k context on a Strix Halo 128GB (EVO-X2, Ryzen AI Max+ 395).
Setup is a fork of Gufo with UD-IQ4_XS quant, offloading some GEMMs to the NPU while the GPU handles the rest. Output verified identical to GPU-only build across 30 long greedy prompts.
Prefill improved from ~1700 tok/s with GPU only. Pinning the GPU fabric clock at max was required to avoid hard hangs and also gave a speedup by itself.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-use
User reports Qwen3.8-Flash-Next at 5.84 t/s generation and 56.10 t/s prompt processing on an Nvidia P100 16GB.
Setup is llama.cpp with IQ3_XXS quant and 80K context, running on an HP Z440 workstation with Intel Xeon E5-1650 v3 and 64GB RAM.
The model is a 125B MoE with 6B active parameters, partially offloaded to CPU. User notes it is slow but works for non-time-critical tasks like email sorting and calendar management.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 64.9 tok/s aggregate decode and 985.8 tok/s prefill on an RX 7800 XT 16GB.
Setup is Strata 0.1.42 with IQ3_XXS and int8 KV cache, 39.97 GiB of expert weights in host RAM and 5,048 experts (8.19 GiB) cached in VRAM. MTP was enabled with --spec 4.
The 64.9 tok/s is aggregate decode over 54 requests; median per-request was 66.4 tok/s with a 56.1-82.8 tok/s range. Other fresh-prompt prefill readings were 912.9 tok/s at 6,731 tokens, 1,004.6 tok/s at 76,599 tokens, 791.0 tok/s at 116,392 tokens and 509.8 tok/s at 38,790 tokens. Across all 61 tail-skip requests aggregate decode was 52.7 tok/s. An informal comparison gave 50.8 tok/s without route tail skip and 56.7 tok/s with it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-uselong-contextagentic
User reports Qwen3.8-Flash-Next at 86 tok/s single-stream on 2x DGX Spark with TP=2, and 60 tok/s on a single DGX Spark.
Setup is vLLM with NVFP4 experts and MXFP8 dense layers, 262K context, MTP with 4 probabilistic drafts, and a retrained drafter. Aggregate throughput at 8 concurrent requests is 195 tok/s on 2x and 137 tok/s on 1x; with thinking off on 36 coding prompts it is 113 tok/s single-stream and 317 tok/s aggregate on 2x.
User's own requant beats the publisher's vLLM image (51.8 tok/s c1 defaults, 96.6 tok/s c8) by 1.67x to 2.02x. IFEval strict 87.4% vs 87.1% shows no quality loss. Two independent 1x replicas (DP=2) reach 241 tok/s at c16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B (Swift-1.5 fine-tune, IQ3_XXS GGUF) at 70 t/s on an RTX 4080 16GB with 81920 context, using MTP plus n-gram speculation.
Setup is llama.cpp (llama-server CUDA build in WSL2/Debian) with q8_0 KV cache, flash attention on, all layers on GPU, and 14.9 GB of the 16 GB VRAM used.
With n-gram only the user got 45 t/s at 204800 context. They note MTP was worth the tradeoff and could grow context further by dropping MTP or allowing spillover into system memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 35B at Q6 running at 25 t/s with about 128k context on 2x Tesla P100 16GB.
User says the model cheats heavily on their personal benchmark (building a clock in Lua for Conky) and still has trouble getting it working.
User asks whether their prompting is the problem or if they are jaded by frontier models, and whether to have Qwen push work to Opus/Sol.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next 125B-A6B at IQ3_XXS running at the full 262,144-token context on a single RTX 3090 with 32 GB of DDR5, using the Strata engine with int8 KV cache and MTP speculation. On a 121,566-token prompt the run reached 1325 tok/s prefill and 59-62 tok/s decode; a 249K-token prompt gave 1008 tok/s prefill and 41.8 tok/s decode with 90.5% expert-cache hits, and short-context decode climbs to 63-100 tok/s.
Setup uses --resident-cpu-experts with a 21 GiB resident budget, 12.2 GiB of experts in VRAM, 21 GiB locked in RAM and 6.6 GiB left on disk, covering 83% of routings without disk reads. The process is pinned to P-cores with taskset -c 0-15 and --pool-affinity p-cores; --prefill auto:16384 raised prefill from 1014 to 1325 tok/s on the 121K prompt. --aux-cpus gave no gain on this chip (60.1 vs 59.6 tok/s decode).
The user asks whether these numbers are good and what others get on comparable hardware, and notes the KV cache is about 3.1 GiB at 262K because only 12 of 48 layers keep growing KV.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at about 28 tokens a second on an M5 Max Mac Studio, run with oMLX and Splash.
The user's complaint is verbosity rather than throughput: the 27B thinks for 10 minutes and generates far more tokens than the Qwen 35B MoE, taking 20 minutes on a task the 35B finishes in 2 minutes, across all harnesses tried.
The user asks whether others see the same behavior.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 200 tokens/second on a DGX Station (ASUS Pro ET900N G3).
No engine, context length or KV cache settings are stated; the run is described as untuned so far.
The user notes the machine's electricity consumption as a caveat.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports about 7 t/s running Qwen3.8 27B Uncensored on an RX 7900 XT 20GB with 32GB DDR5 and a Ryzen 5 7600X.
Setup used Hermes with LM Studio as a server, Q8 KV cache to fit the model in 20GB VRAM; the initial Hermes default configuration offloaded most work to CPU.
User says the experience was poor, with overthinking loops and slow responses, and asks whether the hardware is the problem or their configuration.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 15 t/s on 2x RTX 3060 12GB with 64GB RAM, compared to Strata's 35 t/s on thinking and 50 t/s on code.
Setup is vanilla llama.cpp with IQ3_XXS and no MTP drafter, as the only available MTP Q4_K_M drafter is 3.5x larger and slows decode.
User asks for a working llama recipe, noting tensor split fails with a cudaMalloc error on device 0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next running locally on an AMD RX 7900 XTX at 500,000 context, with decode of 105–160 tok/s.
Setup is Strata with a Q2 quant; 66.4 GB of weights, 23.8 GiB resident in VRAM, 31.6 GB of expert weights and a 28.8 GB PLE table in system RAM.
Context held 506,849 tokens. Quality held between 81% and 84% with thinking off and 88–94% with it on across 4k, 64k and 256k. Decode fell 18% over that range. User calls it an incredible improvement over llama.cpp and Swift 1.5 27b dense, and now uses it as a daily driver.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User benchmarks Qwen3.8 27B at 70 tok/s average on an RTX 3090, running in llama.cpp with 119K context and MTP enabled.
Setup is llama.cpp with MTP speculative decoding and 119K context.
The user also tested Qwen3.8 Flash-Next in Strata at 128K context, averaging 120 tok/s, and GPT 6.1 Sol (High reasoning) which one-shot the prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports bit-exact speculative decoding on Qwen3.8-27B at 106.6 t/s on short prompts on one RTX PRO 6000 Blackwell Max-Q.
Setup is SGLang with BF16 weights and BF16 KV cache, DFlash 8 draft tokens, deterministic inference, Triton attention backend, and three patches including a DeepGEMM fix and PR #42145.
Exact decode is 3% behind vLLM on short prompts, ahead at 2.2k tokens (73.9 vs 67.4 t/s), and 24% behind at 128k (42.6 vs 56.1 t/s). Prefill is slower: 68.5 s to first token on a 128k prompt versus 44.3 s on vLLM. Deterministic mode without the PR was 18.0 t/s, against 26.4 stock.