Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-uselong-contextsummarizationcoding
User reports Qwen3.8-Flash-Next IQ3_S at 81.9 tok/s in calibration and 85.4 tok/s in chat on an RTX 5070 Ti 16GB with 64GB RAM.
Setup is Strata with IQ3_S GGUF, int8 KV cache, 256K context, KV streaming with 32768 resident cells, and MTP speculation. The model is 125B-A6B with 46.84 GiB of experts loaded and an 84GB download.
Two concurrent 256K slots exceeded a 58 GiB container limit during startup with and without batch MTP, and earlier unrestricted runs caused global OOMs and a host freeze. The user settled on one active request at 256K with KV streaming.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next IQ3_XXS at 193.8 t/s generation and 4627.2 t/s prompt processing on an RTX 5090 with 128GB system RAM.
Setup is Strata 0.1.40 with INT8 KV cache, 4096-token context, MTP speculation with --spec 4, and CPU expert-pool offload. The IQ3_XXS GGUF is 75.84 GB with 22.34 GiB of GPU expert cache.
User compares three quants: IQ3_XXS (193.8 t/s at 4K), IQ3_S (161.0 t/s), and UD-IQ4_XS (116.3 t/s). Accuracy screening on 14 MMLU-Pro questions scored 12/14 for all three. IQ3_XXS had the highest median generation speed in the 4K test.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 31.2 t/s decode on a single Tesla P100 16GB with PXA v3.1 Overdrive mode, using a 32 GB quant.
Setup is PXA with a 32 GB quant, experts cached on card and rest streamed, single request.
Overdrive off gives 20.0 t/s; two P100s reach 38.2 t/s, one V100 35.4 t/s, two V100s 55.7 t/s. Against Strata on the same P100, prose decode 31.1 vs 27.0 t/s, code 34.3 vs 34.1, prompt processing 4k 326 vs 226 t/s and 8k 338 vs 246 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next-GSQ-RCO-Abliterated at IQ3_S running with 12GB VRAM and 32GB system RAM, achieving 20-30 tok/sec decode and 300 to ~90 thousand tok/sec prefill on a custom fork of Strata. The Q2 quant achieves 39-45 tok/s decode. The fork has many experimental architectural changes. User compares the experience to local Opus on sub 2k compute.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next 125B-A6B at 20.14-21.13 tok/s on an RTX 3060 12GB with 16GB DDR4 RAM, using a custom llama.cpp build with --moe-direct-io prefetching.
Setup is llama.cpp with IQ2_XS quant and a 68GB model file, run under a 6G cgroup memory limit; the model is streamed from NVMe SSD with no expert pruning and bit-exact output.
Stock llama.cpp on the same machine gives 1.41-2.12 tok/s with 1,140-1,565 major page faults per token; warm cache runs reach 24+ tok/s, cold starts begin at 3.5-4.5 tok/s, and prompt processing remains slow.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports a 2.3bpw fusion of swift-1.5-uncensored and mirai's 2.5bpw model running on RTX 3080/3080 Ti cards at 65-70 t/s with MTP on evals.
Setup is LlamAmpere with 4/4 KV cache, 12GB VRAM, and MTP enabled. The model retained 85% of BF16 performance on LiveCodeBench across 7 runs on 3060/3080/3080 Ti cards.
The user notes the model is usable for standard agentic tasks and subjectively places it between BF16 versions of Qwen3.5 and Qwen3.6. The 3/3 and 3/2 KV cache variants support 205K and 230K context respectively at 11GB VRAM.
User reports running Qwen3.8 Flash-Next on a Strix Halo 128GB at 1600-1800 t/s prompt processing and 60 t/s generation for agentic work. The source discusses Halogen and Gufo with Q4_K_XL but does not unambiguously attribute this Flash-Next speed range to one engine/quant combination; structured engine and quantization are therefore unspecified. The 128GB is unified memory on the Framework Desktop, not additional discrete VRAM.
A separate single CMP 170HX 64GB (unlocked from 8GB) runs two concurrent Qwen3.8 27B sessions with 200k bf16 context each, using DFlash 2 speculative decoding. Aggregate performance at low context is 2000 t/s prompt and 150-300 t/s generation. A 4-hour session with 7.2M tokens in and 1.1M tokens out averaged 1747 t/s prompt and 100 t/s generation aggregate across the two 27B sessions.
The CMP 170HX sits on a USB4 eGPU dock limited to PCIe 2x4. The user dropped cloud usage entirely with this setup, which draws about 400W fully loaded and cost around $3000 total.
Hardware attribution corrected: the Flash-Next figures above belong to the Strix Halo. Two concurrent 27B sessions on the separate CMP 170HX are not two GPUs; the author cancelled the second-card purchase. The CMP figures remain a separate workload and are not assigned to this Flash-Next report.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Heretic (UD-IQ4_XS) at 55 tok/s on code and 50 tok/s on prose on an RTX 4080 16GB with MTP 2 draft tokens.
Setup is llama.cpp b11457 with a 15K-token prompt, 32K context, MTP 2 drafts, mean of 3 fixed seeds. Without MTP the same config gives 29 tok/s on both code and prose.
User notes 2 draft tokens beat 3 (68 vs 56 tok/s on code, 55 vs 36 on prose at 40K), 48K vs 40K produced identical tokens but was 22% slower, and VRAM spill from the desktop dropped code decode to 68, 51 or 42 tok/s depending on 1.2, 1.4 or 1.9 GB of shared VRAM. MTP head precision barely matters (q6_K vs IQ3_S: 83% vs 83% acceptance on code, 58% vs 54% on prose). The 12GB and 24GB context numbers are computed from llama.cpp buffers, not measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writingrpagenticvision
User compares Strata and Infernix running Qwen3.8-Flash-Next Uncensored on a single RTX 5090 32GB with 192GB system RAM on Windows 11. Infernix NVFP4 reaches 137.0 tok/s decode and 3,497 tok/s prefill at 512k context (YaRN 2), while Strata Q4_K_S gets 126.3 tok/s decode and 3,045 tok/s prefill at the same context.
Setup uses int8 KV cache, MTP spec-4 with lm-head-draft, and a 52GB n-gram volume mmap'd from NVMe with 63.3 GiB of experts pinned in RAM. Decode was measured at roughly 39k effective context, with ±10-15 tok/s run noise.
User reports Infernix is 23% faster at 262k and 9% faster at 512k, calls it the daily driver, and notes vision works on both engines.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 665 tok/s structured and 354 tok/s prose on an RTX 5090 + RTX 5060 Ti.
Setup is Basalt, a Strata fork for Blackwell, with IQ3_XXS weights, 64k context, MTP enabled, 400 W, on a 9950X with 32GB DDR5 RAM.
A single 5090 gives 585 struct and 316 prose, about 12% slower, with prefill unchanged. Concurrency of 8 streams totals 623 tok/s. IQ3_S prose is 300 tok/s at ~89% top-1 agreement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Qwen3.8-Flash-Next, a 125B-A6B MoE, on a 128 GB M4 Max MacBook Pro with a new unofficial MLX engine called strata-mlx that reads Strata's GGUF files directly.
Setup uses the model's built-in draft head for speculative decoding. The Q2_0 file is 66 GB and the IQ3_S file is 84 GB. On a Mac capped to 16 GB of memory the 66 GB file runs at 6-7 tokens/second, on a 24 GB machine at 19-23, and on a 32 GB machine at 28-47, with experts streamed from SSD. A 1,537-token prompt runs at 273-287 tokens/second in that mode.
The user notes a 48 GB Mac holds the Q2_0 file whole and never touches the disk, and that speed is not guaranteed on smaller machines.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 140 tok/s on code generation on a single RTX 3090, using a custom CUDA megakernel that runs each speculative-decoding cycle in one launch.
Setup is the megakernel with Q4_K_M weights and fp16 KV cache up to about 80k context, then q8. The same hardware with llama.cpp and MTP gives 73 tok/s on code, 56 tok/s on coding with a 4K-token file in context, and about 1,100 tok/s prefill.
The megakernel is 1.4-1.9x faster than llama.cpp depending on task and context length, and is built for this model on 3090s only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Swift 1.5 at almost 100 tok/s decode on 2x RTX 3090.
Setup is vLLM with an 8-bit AWQ + GPTQ W8A16 quant, 128k context and FP16 KV cache, MTP=5, around 21.6 GB used per card.
The quant was made on 4 RTX 3090; the user says it is less of an overthinker than the original Qwen3.8 27B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 15.8 tok/s generation and 140.0 tok/s prompt processing on an M3 Max 64GB.
Setup is oMLX with an oQ4e quant, Lightning MTP on, MoE expert offload on and 50% resident experts, running a 99GB model at up to 130k context.
Figures come from a 13-request session with PI: 321,309 total prefill tokens, 292,209 cached (90.9% cache efficiency), so the prompt rate excludes cached tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running Qwen3.8 Flash-Next IQ3_XXS at 59.4-65.2 tok/s prompt and 66.6-68.3 tok/s generation on 4 BC-250 boards with 100K context.
Setup is llama.cpp with Vulkan and RPC over 2.5gb ethernet, using speculative decoding with the model's own MTP draft head. The cluster also runs two instances of Qwen3.6 35B A3B at 145 tok/s generation and 450 tok/s prompt with 256K context.
Power draw is 600-700 W for Flash-Next and 800-900 W for the two 35B instances. Boards cost under $100 each.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 29.80 t/s generation and 1193.06 t/s prefill on two AMD Strix Halo 128GB hosts.
Setup is Gufo with two-host RDMA tensor parallelism over USB4 RoCE v2, full Q8_0 GGUF, 262144-token configured context, one session, speculative decoding off.
Measured prompt was 2040 tokens with no cached prefix and 128 predicted tokens; the 262144 context is capacity, not a benchmark length. Source-reported third-party measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8-27B at about 304 t/s decode at 8K context on one RTX 5090, with 221 t/s at 60K and prefill about 12,500 t/s at 60K.
Setup is a vLLM main build newer than 0.27.1 with a custom NVFP4 checkpoint, DFlash2 drafter with 6 speculative tokens, fp8 KV cache, max model length 195,000 and GPU memory utilization 0.92.
Two concurrent slots get roughly 255-299 t/s end to end. An earlier vLLM 0.27.1 setup with qwen3_5_mtp and 3 speculative tokens reached about 123 t/s at 8K and 90 t/s at 60K, with aggregate throughput rising from 67 t/s at one stream to 579 t/s at 16 streams. NVFP4 has a small quality cost versus GGUF Q4_K_M (q_avg 92.5 vs 93.2; HumanEval 89.6 vs 94.5). The GPU draws roughly 380-520 W under vLLM versus about 120 W with llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8 Flash-Next at 42.7 t/s generation and 1294.2 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB.
Setup is Strata at UD-IQ4_XS with speculative decoding on, one request.
tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at 22.2 t/s generation and 325.3 t/s prefill on an M2 Ultra 192GB.
Setup is AX Engine at MXFP4 / affine 8-bit mixed, speculative decoding on, one request.
Medians of 20 trials in defai-digital's AutomatosX benchmark of 2026-10-08. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 35 t/s generation and 1701.6 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB.
Setup is UD-IQ4_XS with speculative decoding on, one request.
tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.