Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks whether a Qwen3.8 Flash-Next run seen in another post (IQ3_XXS, 100-130 t/s decode, 2500+ t/s prefill, 262k context, Strata/Hermes on a single 24GB RTX 5090 Mobile with 64GB RAM) could be replicated on Kaggle's free tier.
The referenced run is not the user's own measurement; the user only asks about feasibility on Kaggle's 2x Tesla P100 16GB or 2x T4 16GB with about 30GB system RAM.
No throughput was measured by the user, so no generation or prefill figure is recorded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B (Swift-1.5 fine-tune, IQ3_S GGUF) at 106.96 t/s generation and 1831.68 t/s prompt processing on an RTX 5070 Ti 16GB.
Setup is llama.cpp with Q8_0 KV cache, 60000 context, flash attention on, and MTP plus n-gram speculative decoding (draft-mtp,ngram-map-k4v, draft-n-max 3).
User is happy with the speed but wants a larger context window and asks for advice on the launch arguments.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B NVFP4 at 143.5 committed output tok/s on two Tesla V100-SXM2 16GB cards with an 85,000-token coding input.
Setup is NInfer-V100X2 with NVFP4 weights, INT8 group-64 KV cache, MTP4, 180,000-token context capacity, PCIe P2P and no NVLink. Prefill averaged about 1,310 tok/s over the 85K input, with peaks around 2,500 tok/s possible under PP2 pipeline parallelism.
Short structured JSONL generation with DFlash7 reached 253+ tok/s, averaging 255.3 tok/s over the final two runs with 118 input tokens and 99.05% draft acceptance. The 143 tok/s route uses an experimental attention implementation that can change outputs and is off by default; on 104 GPQA-Diamond questions with 85K padding it scored 90/104 versus 88/104 for default attention.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Qwen3.8-Flash-Next (125B MoE) on an RTX 4070 Super 12GB with 32GB system RAM, achieving 33.2 tok/s decode and 947 tok/s prefill at 8K context with the Gyro-M variant.
Setup uses Strata with local source patches, GGUF weights, Q4 KV cache with 32K tokens GPU-resident, and MTP disabled. Experts are tiered across VRAM, 16 GiB of RAM, and SSD streaming.
Gyro-S variant reaches 40.8 tok/s decode and 879 tok/s prefill at 8K, with vision support. Benchmarks cover 8K to 256K context; decode averages 3 cached runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticvisionlong-context
User reports running Qwen3.8 Flash-Next (125B MoE) on an RTX 4070 Super 12GB with 32GB system RAM, reaching 40.8 t/s decode and 879 t/s prefill at 8K context for the Gyro-S variant.
Setup is upstream Strata with source patches, Q4 KV cache with 32K tokens GPU-resident, 256K context window, experts tiered across VRAM, 16 GiB in RAM and the rest streamed from SSD, MTP off.
Gyro-M at 8K measured 33.2 t/s decode and 947 t/s prefill; Gyro-S is faster across the sweep, up to 41.9 t/s decode at 128K. Raising prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain. Vision and MTP both reduced speed and were left off Gyro-M.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports running Qwen3.8 27B (4-bit, ~16GB) in LM Studio on a MacBook Pro M5 Max with 48GB unified memory, at around 10-15 tokens/sec.
Setup is LM Studio serving an OpenAI-compatible API to the Pi Coding Agent, which built a React + TypeScript dashboard from scratch over a few hours.
The user says local coding agents are capable but not a drop-in replacement for Claude Code or Codex with frontier models, citing slower inference and reliability concerns.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingmultilingual
User reports Qwen3.8-27B at 5.1-8.3 t/s generation and ~74 t/s prompt on a Radeon 780M laptop with 32GB unified RAM.
Setup is llama.cpp with IQ3_S quant, q8_0 KV cache, 32768 context, and DFlash2 speculative decoding.
User also ran Gemma 4 26B-A4B Q4_K_M at 21.8-25.8 t/s, Qwen3.6-35B-A3B IQ4_XS at 24.4-33.3 t/s, and LFM2.5-24B-A2B Q4_0 at 38.7-40.3 t/s. User plans to use Qwen3.8-27B for coding and Gemma 4 26B-A4B for other tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 27B at 19 tok/s with speculative decoding on an RX 6900 XT 16GB.
Setup is LM Studio with IQ4_XS weights on the Vulkan backend, single GPU, 32GB system RAM.
Without speculation the same model and quant runs at 11 tok/s; a Qwen 27B Q3_K_M run gives 8.4 tok/s and a Qwen 4B gives 51 tok/s with thinking off and 38 tok/s with it on. User asks whether Linux/ROCm would beat Vulkan on Windows and whether these speeds are normal for the card.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next IQ3_XXS at ~27 tok/s decode and ~475 tok/s prefill on an RTX 3060 Ti 8GB.
Setup is Strata 0.1.44 on CachyOS with 128K context, running on an i7-4930K with 50 GB DDR3; the model is partially offloaded to system RAM.
About 7 s to first token per turn. A second GPU (RTX 2080 Ti) is planned but not yet installed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 20 t/s on 8GB VRAM plus 32GB DDR4 RAM with Strata at 100k context.
Setup uses the GSQ-RCO-IQ1_M GGUF quant with a 5GB MTP model, on a custom HIP build for gfx1030 that the Strata dev does not support.
The user says the speed is close to llama.cpp with 35B models, but the model never stopped analyzing a Plymouth boot theme script and burned through 100k context plus a compaction and another 100k context without finishing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writing
User reports Qwen3.8-27B at ~150 tok/s on code at short context on a single RX 9070 XT.
Setup is ExLlamaV3 with EXL3 3.0 bpw quant, DFlash2 speculative decoding, and TabbyAPI, running in a container with RDNA4 patches. Default context is 96k, with 128k available if nothing else uses the card.
At 120k context code generation drops to ~105 tok/s, prose to ~60 tok/s. Prompt processing is ~1,100-1,250 tok/s on short prompts and ~900 tok/s at 128k. Up to 3 concurrent requests share the context. GPQA Diamond scores 88.4% at xhigh effort and ~81% at medium.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at approximately 140-155 tok/s on 2x Radeon AI PRO R9700, up from 31 tok/s on Windows.
Setup is the Radiance engine with FP8+IQ4R weights and MTP speculative decoding at 200K context.
The user also reports Qwen3.8-27B at about 42 tok/s on one R9700 with Vulkan and MTP, Ling 3.0 Tiny at about 183 tok/s on an RTX 5060 Ti 16GB with Q8_0, and Gemma 4 12B at about 90 tok/s on the same card with a QAT UD-Q4_K_XL build and an MTP drafter.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~62 t/s decode and ~1400 t/s prefill on a single RTX 3090 with 8-channel DDR4 server RAM.
Setup is Strata with IQ3_S GSQ-RCO quant and 256K INT8 KV context at a 220W power cap.
User also mentions GLM 5.3 Flash "Maya L" 156.3GB running on a single Tesla V100 32GB via Project Maya, but gives no throughput figure for it. User asks what benchmarks to use to compare the two models on quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 180 t/s on a pair of V100 16GB cards without NVLink.
Setup uses NInfer with NVFP4 weights, Q8 KV cache and MTP 4/0 speculative decoding.
The user calls it the legend of V100 and links the tuxKOH/ninfer-V100X2 repo.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next IQ3_S at approximately 90 tk/s decode on an AMD R9700 32GB with 64GB RAM.
Setup is Strata with IQ3_S quant.
The user notes this is about 3x faster than a 27B model and was surprised by the quality given the small Q3 quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 Flash-Next at 50-70 t/s single-user on dual A100 80GB, dropping to 10-15 t/s with 2-3 concurrent sessions.
Setup is llama.cpp with UD-Q4_K_XL and Q8_0 KV cache, 7 parallel slots at 262k context each (1835008 total), Q8_0 MTP head with draft max 2, mmproj enabled, flash attention on, layer split, 32GB cache RAM.
User is frustrated with multi-user scaling and asks whether to switch to vLLM or run Qwen3.6-35B-A3B per card; use case is long-session agentic coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 57.4 t/s generation on Strata, versus 18.84 t/s with llama.cpp (Unsloth fork) on the same Hermes session.
Setup is Strata with UD-Q4_K_XL GGUF weights, FP16 KV cache and 130k context; llama.cpp ran the same files at 110k context. Strata read a 38,968-token prompt in 23 seconds where llama.cpp took about 10 minutes, and the user reports roughly 3x the generation speed.
The user notes expert cache hit was 94.5% with 1.3% of routed experts over PCIe, and that earlier builds gave only about 45 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next 125B-A6B at a median 116 t/s decode on an RTX 5090 32GB.
Setup is the strata-nvfp4 fork with NVFP4 experts and Q8_0 dense weights, int8 KV cache, MTP draft head with spec 5, and 63 GiB of experts pinned in system RAM.
Decode was median 116 t/s, mean 103, peak 171; prefill averaged 3,332 t/s with a 6,623 t/s peak. The 33m 45s run produced 124,443 output tokens over 87 steps with a 99.0% cache hit rate and no compactions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcoding
User asks whether to switch from a Qwen3.8 27B Swift-1.5 Q6_K setup to a Qwen3.8 Flash-Next Swift-1.5 IQ3_XXS GSQ RCO quant for agentic work.
Current setup runs at 262144 context with Q8/Q8 KV cache on 40GB VRAM across two Radeon VIIs plus an RTX 3080 Max-Q, with 64GB DDR4 RAM, using ROCm and Vulkan. It achieves around 450 t/s prefill at 16384 tokens and 15-30 t/s output with dflash2 speculation.
User is unsure whether the lower-quant Flash-Next would be smarter at comparable speeds, and notes that a mixed Vulkan/ROCm backend rules out engines like Strata, likely requiring a custom llama.cpp fork.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 4bit running at around 40 t/s on an RX 7800 XT 16GB with 32GB RAM, at a max context of 120k.
User is considering upgrading to a used RX 7900 XTX for €900 to reach 250k context and replace AI subscriptions, but finds the current quality not great.
User is not interested in Qwen3.8 Flash-Next.