Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 70 t/s sustained on a 2x RTX 3090 build.
Setup uses the Q4 quant from the syv ais repo and config, with 64GB DDR4 4000MHz system RAM.
The figure is described as sustained throughput; the user offers more details on request.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 39 t/s on dual RTX 3080 20GB GPUs.
Setup uses Q4 quantization with 180k unquantized context on a home server built for just under $2000.
The user notes the cards have coil whine during decode and that the motherboard could support up to 4 GPUs at 16 lanes or 7 GPUs at 8 lanes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-context
User benchmarks five Strix Halo llama.cpp forks with Qwen3.8 Flash-Next on a Flow Z13 (Ryzen AI Max+ 395, 128 GB, gfx1151).
halogen 0.5.4 at 70W reaches prefill 1192/1350/1315 t/s at 8k/32k/131k and decode 33.2 t/s serial and 43.1 t/s drafted. Nathan's toolbox v0.7.4.1 on Vulkan reaches 26 t/s serial and 34.3 t/s drafted, with prefill 539/433 t/s at 8k/32k. myhacsint reaches 25.7 t/s serial and 35.6 t/s drafted, with prefill 532/584/538/427 t/s. strix-llama was not run; the thread reports 14 t/s serial and prefill 532/397 t/s.
Speculative decoding fails the byte-identity gate on every GGUF build tested, with 3/5 to 9/10 prompts differing at temp 0. Serial reruns are byte-identical 10/10. The halogen 0.5.3 changelog claims 1246/1424/1358 t/s prefill at 8k/32k/131k. The model is Qwen3.8 Flash-Next; parameter count is not stated.
User reports Qwen3.8-Flash-Next-UD-Q5_K_XL on Strix Halo 128GB with mmproj, MTP, a 64000 Q8 KV cache and 2GB cache-ram, leaving only 1-2GB free memory.
Q4_K_XL (77GB) and Q5_K_XL (108GB) were also tested. MTP adds 5.5GB even when the file is 2.8GB. F16 KV cache uses 1GB per 10000 tokens; Q8 uses 1GB per 20000 tokens.
Perplexity results: Q5_K_XL Top-1% 94.795, Mean KLD 0.018175, 99.9% KLD 0.591240; Q4_K_XL+50GB-NGRAM Top-1% 93.786, Mean KLD 0.027853, 99.9% KLD 0.948222; Q4_K_XL+25GB-NGRAM Top-1% 93.285, Mean KLD 0.033378, 99.9% KLD 1.090875. The F16 base-kld could not be run due to insufficient memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8-22 t/s on a 32GB M4 MacBook Air with 21GB of allocations.
Setup is a custom inference engine called Cherenkov that combines predictive expert streaming with optional mixed-precision execution, keeping a bounded working set of experts in unified memory rather than loading the entire model.
A one-layer lookahead predicts which experts will be needed next and initiates SSD reads.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next on a Raspberry Pi at 1.95 t/s prefill and 0.44 t/s generation.
User is happy with these speeds despite acknowledging most would find them unusable, and would like a DGX Spark but values the Pi's always-on nature.
User asks about web-surfing solutions for local agents, preferring lightweight options without MCP, and mentions textweb html2md.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
summarizationlong-context
User compares self-hosted Qwen3.8-27B Q8_0 on a RunPod 48GB GPU tier at about $1.22/hr against OpenAI GPT-5.6 Sol for a 47k-token transcript analysis task.
Prompt processing runs at about 1073 t/s and generation at about 17.2 t/s in llama.cpp, which warns about unused NextN/MTP tensors (blk.64.nextn.*), suggesting speculative decoding is not active.
User asks whether switching to Q4_K_M or Q5, reducing context to 64k, enabling MTP, using vLLM, or using a 5090, L40S, A100 or H100 would help.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports best-case MTP decode of 218 tok/s with 99.2% acceptance on a Volta V100.
Setup is the custom NInfer inference engine with software NVFP4.
Throughput is about 200 tok/s without context-copy speculation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports token generation improving from 20 t/s to 49 t/s with the flashnext-e06 fork on 2x RTX 3090, an Intel Ultra 7 270k Plus, and 192 GB DDR5@5600 MHz.
Setup uses an MTP draft model in Q8_0 and ngram-mod speculative decoding.
Prompt processing is faster on the main branch.
User reports Spomin, a router for live KV cache compaction using a llama.cpp fork, tested exclusively with Qwen 3.8 27B as the main LLM.
Worker models are a fine-tuned Qwen2.5 3B and a Qwen3.5 9B with larger context, with DFlash2 for speculative decoding.
On a research benchmark the score improved from 31 to 37/100 with the larger worker versus 57 unmanaged, with 75.2% less primary input and 60.7% less active time. On Tetris the run scored 16/16 plus 11/12 checks in 69.8 min versus 16/16 plus 12/12 in 149.6 min, with 22.6% fewer tokens. A 10:1 compression example goes from 400k source tokens to 40k summary tokens and 180k resident tokens. No GPU is specified and no t/s figures are given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writing
User reports Qwen3.8 Flash at 18.7 t/s generation on a Ryzen 7700 with 64 GB DDR5 and an RTX 4060 Ti 8 GB.
Setup is a custom llama.cpp fork (myllama.cpp) running the 35B MoE model with 6B active parameters, CUDA and Windows specific, with no MTP support.
An initial IQ4_XS quant gave 100 t/s prefill and 20 t/s generation, degrading to 15 t/s at 100K context. A switch to Q5_K_M dropped to 15 t/s prefill and 10-13 t/s generation from MMAP page faulting. A four-layer system followed: an unbuffered raw file reader with a RAM buffer for 2 slabs reached 150 t/s prefill, a RAM expert cache with decay factor (55-56 GB, covering 85-95% of requests) gave 14-15 t/s generation, memcpy from cache to buffer during prefill reached 215 t/s prefill, a 2 GB VRAM cache for top experts gave 18.0 t/s generation, and direct IO for embeddings gave 18.7 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User compares Qwen 3.8 27B at Q6_K and about 31 GiB resident against Qwen 3.8 Flash Next at UD-Q4_K_XL and about 86 GiB on an ASUS ROG Flow Z13 with Ryzen AI Max+ 395, 128 GB unified memory, running Arch Linux. Decode is 10-15 tok/s.
Setup is the llama.cpp backend with the LlamaStash tool and the Pi coding harness. Cold prefill runs 31k tokens in 3 min (about 172 t/s) and a 128k window in about 18 min (about 118 t/s). Warm follow-up turns take about 45s. MTP gives 7.3 to 22.4 tok/s on an empty window and 1.15x at the full 256k.
Flash Next completes the same 5/5 coding tasks with 45% fewer tokens, 76.5s against 289.8s for 27B. On the Artificial Analysis index Flash Next scores 40 against Opus 4.8 at 42, and 27B xhigh scores 34 against Opus 4.6 at 32. Thinking is 90-95% of generated tokens. The setup costs $0/month and runs fully offline.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports slow performance with Qwen3.8-Flash-Next on an M2 Max 96GB using the oMLX engine. Benchmarks give 124.1 pp TPS and 20.4 tg TPS at pp1024/tg128, 170.7 pp TPS and 16.7 tg TPS at pp4096/tg128, 214.3 pp TPS and 18.5 tg TPS at pp8192/tg128, and 235.7 pp TPS and 18.3 tg TPS at pp16384/tg128.
Setup is oMLX, a custom MLX-based engine, with 32768 context and a turboquant_kv_bits 8.0 KV cache. The user also tried oQ4e-fp16-mtp, oQ3-fp16-mtp and oQ3-MTP variants.
The fp16 variants are even slower at roughly 150 pp TPS and 8 tg TPS. The user notes llama.cpp achieves 350-400 pp TPS. The model is MoE with 6B active parameters.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports Qwen3.8-Flash-Next at ~38 tok/s decode on a 48 GB M3 Max, with prefill at 210-380 tok/s.
Setup is the user's oMLX fork with a custom Metal kernel for ternary routed expert gate/up (Bonsai T5 packing, ~1.875 bpw), Q3 expert down projections and the 53 GB n-gram table left on SSD as Q8 and mmapped per token, about 35.6 GiB resident. The oMLX memory guard is on the 'safe' profile with a 48 GB limit, one model loaded and one request at a time.
A 65,536-token prompt plus 256 output completed at 30.7 tok/s with a 42.3 GiB physical peak, and 96K trips the prefill guard. Against Unsloth UD-Q4_K_XL, which does not fit in 48 GB, MMLU is 83.0% vs 89.7% and KLD vs Q8_0 is 0.49 vs 0.036.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-context
User reports Qwen3.8-27B running on an RTX 5060 Ti 16GB with vision and 85K context, leaving 1.5GB VRAM headroom.
Setup is beellama, a llama.cpp fork, with MTP speculative decoding, the IQ3_XXS-mtp quant from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, and a kvarn4 KV cache quant.
The user notes mmproj could be moved to CPU for more VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 27B at 12-15 t/s generation on an AMD Ryzen AI 9 HX 470 with 96 GB RAM using the strix-halo-llamacpp fork.
Setup is the Nathanw1014 strix-halo-llamacpp fork of llama.cpp, which the user picked after testing several forks.
On stock llama.cpp, Qwen 3.8 27B and Qwen 3.8 Flash both ran at 10-12 t/s with about 50 t/s prompt processing. With the fork, Flash reaches 15-18 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-context
User benchmarks Qwen3.8-Flash-Next-Uncensored on a Bosgame M5 (128GB/2TB) running Fedora 44 with Vulkan driver 26.1.7, reaching 198.99 t/s prompt processing and 21.53 t/s generation at 200,000 context.
Setup is a llama.cpp fork (strix-halo-qwen4exp-b10685) with the Q5_K_M quant and a shared-Q8_0 MTP draft model.
llama-benchy results across context lengths: pp2048 460.38 t/s, tg512 34.86 t/s; pp4096 465.96 t/s, tg512 33.04 t/s; pp8192 437.14 t/s, tg512 31.18 t/s; pp16384 440.89 t/s, tg512 31.21 t/s; pp32768 407.48 t/s, tg512 29.65 t/s; pp65536 351.55 t/s, tg512 27.40 t/s; pp131072 265.54 t/s, tg512 23.87 t/s; pp200000 198.99 t/s, tg512 21.53 t/s. MTP acceptance rate stays at 80% even at context above 200K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports prefill improvements of 2.2-2.5x on Qwen3.8-Flash-Next across 8k, 37k, and 119k contexts on 2x RTX 3090 with dual Broadwell Xeon and DDR4, with decode performance unchanged.
Setup is llama.cpp on a custom branch flashnext-e06, a 150-slot expert cache moved off GPU during prompt processing, 261k context, and an f16 KV cache. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC.
Prefill is 223.7 t/s at 8k (was 99.9), 212.6 t/s at 37k (was 88.1), and 206.5 t/s at 119k (was 81.3). Decode is 34.3 t/s at 8k, 41.2 t/s at 37k, and 33.9 t/s at 119k. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, and 1461s to 575s at 119k. A quality screen showed no regression, and MTP acceptance is 0.79-0.83.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 10-20 t/s on an RTX 3090 at 65,536 context, with a detailed log showing tg=11.44 t/s.
Setup is llama.cpp with the Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf file, a q4_0 KV cache for both K and V, and MTP speculative decoding with draft acceptance 0.4125. Prompt processing runs ~334-338 t/s.
The title says IQ3 XXS while the body also mentions IQ3_S - 3.4375 bpw. More than 1 GB of VRAM is left after loading.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 20 t/s split between an RTX 3060 12GB and an RX 9070 XT, and at 5 t/s on a 780M iGPU with 5400MHz DDR5.
The 20 t/s figure is for the two-GPU split configuration and the 5 t/s figure is for the iGPU.
User prefers the 5 t/s iGPU setup for system usability.