Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a roughly 2x speed boost for Qwen3.8 27B on Apple Silicon using the MTPLX framework.
User also reports Qwen3.6 35B A3B at about 55 t/s decode and about 300 t/s prefill, with a peak of 623 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Next running faster than a dense 27B model on an adapted DSv4 streaming stack.
Setup uses a 3-bit MoE quant against the dense model's 4-bit quant, in low power mode.
No throughput figures are given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 51.06 t/s single-stream and 184.03 t/s aggregate at c6 on 2x DGX Spark in TP2.
Setup is SGLang with QSA, NEXTN and FlashInfer GDN.
Disabling NEXTN gives 26.4 t/s at c1 and 111.8 t/s at c6. Medium thinking scored 22/24 on a 24-task LiveBench-style pilot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 6 t/s on an RTX 5080 16GB with 64GB RAM, and finds it unusable.
Setup is llama.cpp with offloading to CPU/RAM. The first attempt used the IQ3_XXS quant, the second the Q4_K_M quant at 4.27bpw AD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 22.0 t/s generation and 24.96 t/s prompt processing on a single RTX 5070 with 12 GB of VRAM.
Setup is llama.cpp with the IQ1_S quant at 10,000 tokens of context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next (125B MoE + 51B n-gram table) at 32 t/s decode and 463 t/s prefill on 2xRTX 3090 + 96GB DDR5.
Setup is llama.cpp with experts offloaded to RAM and the n-gram table offloaded to NVMe, UD-Q4_K_XL, a 640K serving pool (4x160K lanes, q8 KV), and the 51B n-gram/PLE table mmap'd on NVMe. The MoE expert cache (RFC #24528 branch) at explicit --moe-cache 5000 gives +30% decode, while auto grants nothing. Concurrency erases the gain, so the user fans out shallow and serializes deep. Prefix caching reprocesses ~24 tokens on warm chat turns, with checkpoints anchored at turn boundaries and DeltaNet state included. The user found and fixed a gridDim.y overflow crash (deep multi-stream batches, QSA indexer norm) with a 2-line reorder, the same shape as the pending upstream fix in PR #27941. Serving holds 32 t/s while pulling 90GB via cgroup write-caging (systemd-run --scope -p MemoryMax=6G).
In vLLM, the user grafted the unmerged mmap-PLE PR (#54129) onto the qwen38-flash-next image, serving the table from NVMe under W4A16, TP=2, in 53GB where the official recipe wants ≥110GB free RAM. Real-table decode is 15.3 t/s vs 16.0 with the table stubbed, so the NVMe n-gram costs ~4%. Selective experts-only UVA offload is mandatory: naive --cpu-offload-gb recopies the whole offload every step (2.5 t/s), while --cpu-offload-params routed_experts.*_weight_packed gives 16 t/s. MTP speculation hurts when the PCIe bus is the bottleneck (13.9 vs 16.0). vLLM is currently being optimised and tested for concurrency, and single stream is behind llama.cpp right now.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextagentic
User benchmarks Qwen3.8 27B on an M4 Max 128GB across five runtime and quant stacks at contexts from 32K to 256K.
The stacks are oMLX AWQ 5-bit, oMLX oQ8e, mlx-dspark, and MTPLX 4-bit and 8-bit.
Best decode at 256K is oMLX AWQ 5-bit at 15.2 t/s, while MTPLX collapses at 256K. Prefix caching is crucial for effective speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a fork of NInfer with tensor parallelism and YaRN scaling decoding at 119 t/s with MTP and 57 t/s without at 653k context, and 48 t/s decode at 1M context, around 100 t/s with MTP.
Setup is a fork of NInfer with tensor parallelism and YaRN scaling.
Against vLLM at 42 t/s at 653k, MTP acceptance drops to zero past 262k. Prefill is 1.2-1.3x slower than vLLM. Two GPUs reach 75 t/s versus 54 t/s on one at 250k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 22.30 t/s generation and 1003.85 t/s prefill on an RTX 5070 Ti Laptop 12GB plus an RTX 5060 Ti 16GB eGPU.
Setup is llama.cpp with layer-split across the two cards. Q4_K_XL measures 1003.85/22.30 t/s pp512/tg128, Q5_K_XL 923.21/19.11, and Q6_K 901.18/18.23.
MTP speculative decoding raises generation to 32-38 t/s. The context ceiling is 49,152 for Q6_K and 73,728 for Q5_K_XL. The user also notes tensor-split syntax differences and eGPU power interruption.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports tensor-read-lazy with mmap fitting a Q4 model on a system with an AMD Radeon AI PRO R9700, RX 9070 XT, and Radeon 8060S (Strix Halo).
The model is 103.68 GiB with 176.94 B params.
Prompt processing reaches 279.25 t/s and generation 26.63 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-Flash-Next at 79 GB in 2-bit running at 350K context for 3.5 hours on a 128 GB M5 Max, over 100 turns.
Prefill runs 1,561 t/s at 5.6K context tapering to 318 t/s at 111K, with normal turns at 77-854 t/s from prefix reuse. Decode is about 30-35 t/s at small context down to 11.5 t/s at 169K, with a dip to 7.7 t/s at around 140K due to Low Power Mode. Decode figures are effective throughput with speculative decoding.
The user calls the experience strong for the first ~100K context, then reports role confusion on long-tail tasks, suspecting the 2-bit quant or preview-model long-context quality.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at Q8_0 generating for over 45 minutes because of excessive thinking even at low reasoning effort.
Setup is llama.cpp RPC on 2x Strix Halo 128GB.
User compares the run with Qwen3.8-27B and DeepSeek V4 Flash on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a custom Q4 quant with spliced tensors from Unsloth and AtomicChat quants, with MTP enabled giving +70% decode at 22 btps.
Setup uses custom metal-optimized sparse attention, with prefill reduced to 170 tps at 4K and 150 tps at 256K context.
Q4_0 MTP matches Q8_0 acceptance rates at half RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports prefill of about 4000 t/s and decode of about 60 t/s at 256k context on 4x V100 32GB.
Setup offloads 50GB of ngram data to system RAM.
MTP improves decode at low concurrency.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-usevisionagentic
User reports Qwen3.8-Flash-Next at roughly 110 t/s, ranging from 76 to 125 t/s, on a single RTX Pro 6000 Blackwell with 96 GB, at 165,000 to 170,000 tokens of context.
Setup is vLLM with the NVFP4 quant and INT4 quantized PLE n-grams, MTP speculative decoding, prefix caching, and PLE CPU offload.
The user praises quality for coding and agentic tasks.
User compares Mac Studio configs (M5 Ultra 96GB, M5 Max 128GB, M5 Max 64GB) for local LLM use.
They discuss Qwen3.8-27B at Q8 and Qwen3.8-Flash-Next (6B active) at various quants, including Flash-Next on 96GB Ultra with UD-IQ4_XS (93.7GB, 91.1% retention) and on 128GB Max with UD-Q4_K_XL (111GB, 93.5% retention), with the N-gram layer offloaded to SSD.
User asks for advice on bandwidth versus quant tier, and whether to just get the 64GB box. No actual benchmark numbers are reported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash Next at 12-15 t/s on an RTX 3060 12GB in a Thinkstation P520 with 256GB RAM, at 65k context.
Setup is llama.cpp server with the model offloaded to GPU at --n-gpu-layers 999 and MoE experts on CPU at --n-cpu-moe 48.
The user notes the model is slow and low context, but output quality is much better than the previous Qwen 3.6 35B A3B. Performance degrades significantly when the system is busy.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 22 t/s with the Q6_K_XL quant on 4x RTX 3090, compared to 47 t/s for Q4_K_XL.
Setup is llama-server with --split-mode layer, --kv-unified, --flash-attn on, and speculative decoding (ngram-mod). Context is limited to 200k to avoid OOM, with no KV quantization.
User mentions MTP for Qwen 3.8 Flash Next may land upstream, and prefers 27B_Q8 as a daily driver.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 49.7 t/s decode on structured output and 34.8 t/s decode on prose with TP2 across two DGX Sparks, plus prefill of about 2,875 t/s for 11k tokens.
Setup is MTP k=3 in eager mode. MTP acceptance is about 3.8 tokens/step on structured output.
A single node reaches about 35 t/s on structured output.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports the AtomicChat build of Qwen3.8-Flash-Next running a full 256K context on 96 GB RAM and 40 GB VRAM, at 22 t/s.
The build splits the n-gram table to SSD. The user previously ran unsloth quants, which crashed at around 60K context and were slower at 12-14 t/s.
The AtomicChat build handles a 217K token conversation without crashing. Prefill takes about 18 minutes for 220K context.