Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 36.1 t/s on an RTX 4070 Ti Super, up from 30.3 t/s after raising Strata's --adapt-every from 1 to 6.
Setup is Strata with UD-Q4_K_XL GGUF on a single card, tested on four Slovak-language chat prompts; the fourth prompt describes 20K tokens of source code.
Per-prompt gains ranged from +12% to +55% with UD-Q4_K_XL; a UD-IQ4_XS run reached 63.1 to 71.6 t/s across the same prompts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
rpcreative-writing
User reports Qwen3.8-27B Defiance-Uncensored decoding at 86-90 tok/s on an RTX 4090.
Setup is llama.cpp b11609 with IQ4_XS at 131K context, with MTP speculative decoding accepting 38-42% of draft tokens; llama-server used 21.4 GB.
The user also checked Q6_K_P at the file level only and did not run it because it does not fit on the card.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 9.86 t/s generation and 82 t/s prompt processing on an Nvidia AGX 64GB embedded computer.
Setup is a recompiled llama.cpp with Gyro format support, running the 108GiB TQ3_0 GGUF at a 50W power envelope.
Other models on the same machine: Qwen3.6-35B-A3B Uncensored Q6_K_P at 23 t/s for one request and 43 t/s for two, Qwen3.8-27B Q8_K_XL at 8.3 t/s with MTP, Qwen3.8-27B Q4_K_M at 10.5 t/s with MTP, and Qwen3.6-35B-A3B Heretic APEX at 42 t/s with MTP. User is weighing a 128GiB Framework Desktop and doubts it is a step up.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Swift 1.5 Qwen 3.8 Flash Next at 19-20 tok/s on a Tesla M40 24GB in a Dell T420 with dual Xeon E5-2470 and 192 GB DDR3 RAM.
Setup is a Polystrata fork for the older CUDA compute capability, with IQ3_S weights; the GPU was limited to 180 watts.
GLM 5.3 Flash at IQ3_XXS ran 3-4 tok/s and the user suspects it was CPU bound, with GPU usage under 40%. Qwen 3.8 27B at UD_Q4_K_S ran 2-3 tok/s with 100% GPU offload.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 41.5 t/s on an RX 6900 XT 16GB, generating a roughly 4500-token animated SVG response; a Python task ran at about 32 tok/s.
Setup is Strata with Q2_0 weights, 64k context and Q8 KV cache, using around 13.2GB VRAM on a Ryzen 7 5700X with 32GB DDR4 3600.
The user has not filled the 64k context yet and wonders whether Q4 KV would allow more; the plan is 64GB RAM with IQ3_S, and IQ3_XXS is also being considered.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-uselong-context
User publishes two EXL3 quants of Qwen3.8-Flash-Next for the sushi engine on Apple Silicon, reporting about 57 tok/s decode for the 4-bit pack and 63 tok/s for the 5-bit on an M4 Max with 128 GB.
Routed experts come from ExLlamaV3 v1.6.0; dense layers, embeddings, vision tower and MTP head are copied from beamster's Sushi-4bpw. Prefill at 64K is about 690 tok/s.
The 4-bit pack is 63.7 GiB and the 5-bit 77.8 GiB; KLD against BF16 is 0.94 for 4-bit and 0.79 for 5-bit. Needles in a 400K-token document scored 4/4 for both, and tool calling 28/28 for the 5-bit. The eval is 24 prompts on one machine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next running at about 40 tokens/sec in a local coding workflow, alongside Qwen3.8 27B, driven from VSCode with Copilot as the harness.
The workflow has the model implement GitHub issues, write tests, run the app, push and open a PR, with Muse or Gemma swapped in for PR review.
User says Copilot has been timing out and needing retries, which is annoying at that speed, and is considering Pi or OpenCode as a replacement harness.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next FP8 with MTP at 450 t/s prefill and 20 t/s generation on 4x Intel Arc Pro B70 (128GB VRAM total).
Setup is vLLM in a container with 128GB system RAM, using the qfn-38-fp8-4xb70-mtp configuration.
The figures come from LLM-run tests and the user notes it is slower than some benchmarks found elsewhere; prior manual vLLM attempts on multiple B70 cards crashed for almost every model until 2^32 overflow fixes were backported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks for a Vulkan/radv-based inference engine that reaches over 1400 tok/s prefill for Qwen3.8 Flash-Next Q4, and reports their own tiny engine gets around 50 tok/s decode but prefill stuck below 1k tok/s.
They suspect the gap is the Vulkan versus HIP/ROCm implementation. No GPU or system memory is named.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at about 86 tok/s decode on 2x DGX Spark, with prefill improved to 2892 tok/s at pp2048 c1.
Setup is a 2x Spark tensor-parallel recipe (v2.2.0) with a 16K cold TTFT of 5.49 s; the 1x release (v2.4.0) decodes about 60 tok/s and prefills 1916 tok/s.
Prefill gains come from reading GDN prefill inputs as views and running one projection as a single BF16 GEMM; decode is unchanged and both releases passed the same quality gate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports peak prompt processing of ~2500 t/s on prompts over 80k tokens and decode of 90-120 t/s with Swift 1.5 IQ3_S on two RX 7900 XTX cards in eGPU enclosures over OCuLink PCIe 4x4, on an HX370 mini PC with 64GB DDR5 5600.
Setup is Strata 1.4.4.1 with int8 KV cache, 262144 max context, MTP speculation, expert cache 7000, 20480 KV resident, 50 GiB resident budget, and MMAP to reduce system RAM pressure; vision is enabled.
The post is mainly a guide to building a tuned HIPblasLt table for the user's own hardware; the throughput figures are given as ranges.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B (abliterated) at ~43 tok/s on an M4 Max 128GB.
Setup is mlx-serve 26.10.2 with MTP enabled; the 27B build is 16GB.
The user also lists other abliterated models in their picker: Smart 35B at ~106 tok/s (24GB), Vision 27B at ~22 tok/s (15GB), Coder 80B at ~78 tok/s (49GB), DeepSeek-V4-Flash 284B at ~26 tok/s (75GB), and Flash-Next 125B at ~90 tok/s (77GB). They ask for suggestions on better packs and MTP or mlx-serve settings for this hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next decoding at a median of 51-59 tokens/s (p90 ~65) on two M3 Ultra 256GB Mac Studios.
Setup is MTPLX with a 4-bit mixture-of-experts model (107 GB on disk) and multi-token prediction, carrying 116-142k-token prompts with a 97-99% prompt-cache hit rate.
The user also compares DeepSeek V4.1 Flash against MiMo V2.6 Flash head to head (8 agents each, 3 hours): 11 vs 7 PRs merged and 1 vs 4 stalls. DeepSeek costs about 1.6x per output token but less per merged PR, so every builder now runs DeepSeek.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 36.5 t/s decode on a single 16 GB RX 9070 XT, using a custom llama.cpp fork with adaptive KV streaming over PCIe.
Setup is llama.cpp with IQ4_XS weights and Q8_0/Q4_0 KV cache, running the full 262,144-token context with the KV cache held in host RAM (~7.4 GB) and streamed in bounded working sets.
Decode falls from 36.5 t/s at 1k to 9.5 t/s at 262k; with MTP speculative decoding it reaches 71.8-86.5 t/s at 1k-4k and 31.1 t/s at 262k. Prefill is 1368.4 t/s at 1k and 585.8 t/s at 262k. The card runs at a 231W limit with a 3.6% prefill penalty and no decode loss.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Qwen3.8-Flash-Next-GSQ-RCO-Abliterated at IQ3_S with 12GB VRAM, 32GB RAM and NVMe, achieving 40-50 tok/sec decode and 1300-1500 tok/sec prefill at 131k context with a 44k thinking budget.
Setup uses a custom fork of Strata with experimental architectural changes, offloading to system RAM and SSD.
The user describes the performance as feeling like local Opus in some regards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 73.2 t/s generation.
Source evidence (verbatim):
Source: https://gist.github.com/PierpaoloPernici/f1d1382f8e357b4faffb1a9f584cc1df
# vLLM vs NInfer: Qwen3.8-27B benchmark on an RTX 5090
Hardware: NVIDIA GeForce RTX 5090 32 GB, driver 610.57.04, CachyOS Linux. Engine vLLM 0.27.1, weights NVFP4 (Qwen3.8-27B NVFP4 for RTX 5090, modelopt), fp8 KV cache, max-num-seqs 16, tool-eval-bench harness, PP=2048 TG=128. Plain run without MTP, cold, no prefix caching, concurrency 1: client decode 73.2 tok/s at context depth 0, 62.7 at 4K, 54.0 at 8K, 40.4 at 16K, 24.4 at 32K. vLLM server-side prefill is about 6,000 to 18,000 tok/s. With MTP speculative decoding @3 at concurrency 1: 117.3 tok/s at depth 0. At concurrency 4 without MTP: 217.3 aggregate tok/s at depth 0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a fully local voice and chat assistant running Qwen3.8-27B (dense, 27.3B, Q8) at roughly 68 tok/s decode and ~1.5K tok/s prefill on 2x RTX PRO 4000 Blackwell 24GB, with an RTX 5060 Ti 16GB handling display and helper models.
Setup is a patched llama.cpp build for Blackwell with speculative decoding (MTP, ~62% draft acceptance) and a vision projector, serving 256K context across 2 concurrent slots. The machine is an AMD Threadripper 9960X with 128 GB DDR5 ECC on Ubuntu 26.04, drawing ~820 W across GPUs and CPU.
The user built a custom Python proxy (~22 modules) in front of llama-server with tool-calling, context compaction, a document workspace, and six MCP servers (filesystem, SearXNG web search, ChromaDB RAG with bge-m3, reminders, documents, sandboxed code editor). A voice pipeline with wake word, faster-whisper STT and Piper TTS is currently paused. The user asks for recommendations and comments.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~150 tok/s generation and ~4k prefill on two RTX 5090s.
Setup is Strata with Q5_K_XL weights.
User is considering selling one 5090 and buying four R9700s for roughly the same price, doubling VRAM, and asks how that would affect which models can be run and their performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 54 t/s decode and 1,100 t/s prefill on a single AMD RX 7900 XT with 20GB VRAM plus 96GB system RAM.
Setup is Strata with Huihui UD-IQ4_XS quant (2/3 experts IQ3_S, rest Q8+) and KV streaming for the full 262k context window, using auto:32768 context on ROCm 10.1.
User is happy with the result but asks for advice on speeding up prefill further.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next IQ2_XS at 12.5 tok/s on an RTX 4060 8 GB with 24 GB RAM cap, up from 5.2 tok/s stock.
Setup is llama.cpp with moe-cache plugin, IQ2_XS quant, 68 GB model streamed from NVMe with CPU-side expert management.
User also reports Qwen3.6-35B at 22.6 tok/s with 12 GB RAM and GPU (6.9 stock), 9.9 tok/s CPU-only (3.1 stock), GLM-5.3-Flash 102 GB at 1.75 tok/s (0.32 stock), Qwen3.5-122B IQ2_XXS at 7.7 tok/s (1.89 stock), and gpt-oss-120b at 4.47 tok/s (0.37 stock). Notes stock is faster when model fits in RAM (42.9 vs 36.6 tok/s).