Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next with DFlash2 speculative decoding on an RTX 5090, reporting 118.63 tok/s single-stream decode.
Setup is TensorRT-LLM with NVFP4 weights, a 90-token prompt and 512 forced output tokens, five trials per batch size.
Aggregate decode throughput rises to 1,284.55 tok/s at batch 12 with the selector policy, while per-client throughput falls to 92.27 tok/s at batch 15. A V2 dual-cache deadlock was fixed before the batch-15 trials completed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next at 57.4 tok/s decode and 586 tok/s prefill on an M4 Max 128GB, where the model fits fully in memory.
Setup is oMLX with the Jundot oQ4e+MTP MLX checkpoint; the 36GB paging configuration uses oMLX 0.7.0 expert offload with resident fraction 0.28 (143/512 experts per layer), PLE on SSD and MTP off.
The same model on an M4 Max Studio 36GB measures 15.55 tok/s steady (n>=1024) and 13.0 tok/s short-run (n=128); exact replay on repeated prompts adds about 40% (14.3 to 20.1 tok/s). Other memory tiers are simulated: 48GB at 192 experts ~15.9 tok/s, 48GB at 128 experts ~11.8 tok/s, M4 Pro 32GB ~9.7 tok/s, M4 24GB ~4.9 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B NVFP4 on an RTX 5090 decoding at 267-282 tok/s on a mixed short-prompt set with the DFlash2 block drafter, against 192 tok/s for the recipe's MTP-10 and 77 tok/s with speculation off.
Setup is NInfer with NVFP4 weights and speculative decoding; the same campaign also measured llama.cpp, Ollama and vLLM, and a 5080 16 GB desktop for comparison.
The 450 tok/s NVFP4 recipe figure reproduced at 426-457 tok/s on a recall workload that copies from the prompt, while ordinary prose decodes at 113-141 tok/s; 8 concurrent requests totalled 673 tok/s with MTP-3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next REAP-288 at ~28 tok/s decode on an M4 Max.
Setup is stock mlx-vlm with MLX 4-bit weights and the full n-gram table resident at about 68 GB; the build keeps 288 of 512 experts per MoE layer.
Decode is runtime-dependent: 28-65 tok/s overall, with ~37 tok/s on pmlx streamed from NVMe and ~65 tok/s peak, ~41 tok/s sustained at 4K on pmlx with the table in RAM. HumanEval pass@1 is 91.5% versus 93.9% for the stock 512-expert Q4 build.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contextvision
User reports Qwen3.8 Flash-Next at 73.8 tok/s on an M4 Max with 128 GB unified memory.
Setup is mlx-serve with Q4 affine routed experts, BF16 core and verification head, native MTP at depth 4, and the full 262,144-token context; the 51.2 GB n-gram lookup table streams from SSD.
Matched warm workloads measured 71-74 tok/s for code and technical prose, with serial decode at 33.2 tok/s and a 2.22x MTP speedup. Long-context prefill reached 280.7 prompt tok/s at 248,445 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathagentictool-usevision
User reports Qwen3.8-27B at 94 t/s on an RTX 5090.
Setup is llama.cpp with IQ4_XS GGUF quantization, single stream, flash attention on, thinking mode sampling.
With MTP speculative decoding it reaches 147 t/s (1.57x) and with DFlash2 167 t/s (1.78x). The model is 13.1 GB at 3.84 bits per weight.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-Flash-Next at 33.7 t/s decode on six RTX 3090s over PCIe without NVLink, up from 10.2 t/s on stock llama.cpp.
Setup is a llama.cpp-multigpu fork with UD-Q4_K_XL GGUF and q8_0 KV cache, 5 slots of 262k context, layer split across the cards.
Prefill rose from 263 to 2111 t/s at 250k context on the 3090s, and 744 to 7403 t/s on six rented RTX 4090s. Five-session decode went from 2.3 to 27.3 t/s per session on the 3090s. The user notes the 10.8x figure is against a 2.3 t/s baseline and should not be quoted. N-gram speculation gave 2-2.5x on code rewrites and 1.5x on code explanation but was off for the benchmarks. MTP was tried and was slower, so it is not included.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-Flash-Next at 38 t/s serial decode and 1,311 t/s prefill on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S) box.
Setup is halogen 0.17.1 with a 262k context, MTP speculative decoding on, and the model fully in unified memory; the same run reaches 64 t/s with MTP and 72.7 t/s aggregate across two concurrent streams.
A llama.cpp ROCm build with qwen4exp MTP reached 47 t/s, and halogen 0.16.2 gave 42-46 t/s with NPU calls 30x slower. The NPU sidecar handles search, dedup, decisions and screening; a keyword-poor task averaged 13.6 min with NPU retrieval against 18.7 min without.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 18.5-19.4 tok/s decode on two RX 6800 XT/6800 16GB cards with layer split.
Setup is llama.cpp via Ollama 0.33.2 with Q4_K_M, 65536 context, MTP speculative decoding (draft_num_predict 2). Prefill is ~230 tok/s on a fresh 15k-token prompt; cached runs showed 3,600-13,000 tok/s. Forced flash-attn dropped decode to 16.4 tok/s.
User asks whether ~58% of theoretical bandwidth (297 GB/s of 512 GB/s) is the expected ceiling for hybrid linear-attention models on ROCm, and notes Vulkan crashes mid-decode with DeviceLostError.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 43.9 tok/s generation and 1817 tok/s prompt processing on an RTX 3090 + RTX A4000 at 32k context.
Setup is Strata with UD-Q6_K_XL GGUF and int8 KV cache, 262144 native context, speculative decoding with MTP, experts offloaded to CPU.
At 200k context Strata Q6 does 48.8 tok/s versus llama.cpp's 10.6 tok/s, and TTFT drops from 595 s to 118 s. Q6 is 12-31% faster than Q8 and 19 GB smaller.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Next at 197 t/s generation and 3,300 t/s prefill on an RTX 4070 Ti Super 16GB.
Setup is Strata on CachyOS with Q2 weights, 10k and 20k token prompts, 15.6GB VRAM plus 44GB system RAM used.
An IQ3XXS run reached 120 t/s generation and 3,000 t/s prefill with 50GB system RAM used; total system power was 350W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at a steady 30 t/s on an RTX 5090 Laptop 24GB.
Setup is llama.cpp with Q4_K_M and 8-bit KV cache at 65k context, using 21 GB of VRAM.
The user also tested Unsloth UD_Q5_K_XL at 27 t/s, and NInfer models reaching 82.54 t/s (8-bit, 132k context), 92.4 t/s (NVFP4, 32k context), and 86.63 t/s (NVFP4, 4-bit, 64k context). The user says the laptop is too slow for coding and recommends a DGX Spark instead.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 55 t/s on an RTX 4080 with 128 GB DDR5 RAM, using Strata with Q3 weights and Q8 KV cache at roughly 120-150k context.
Setup is Strata with the 3060 left unused; adding the second GPU drops generation to 45 t/s and prompt processing to 250 t/s from 590 t/s.
With Unsloth Studio the same model reached 27 t/s on the 4080 alone, 23 t/s with both GPUs, and 16 t/s with both GPUs on an older PCIe 4.0 x1 link. The user asks whether the 3060 is counter-productive alongside the 4080.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 104 tok/s on a mixed 4-GPU rig (RTX 5060 Ti, RTX 3090, 2x RTX 3060).
Setup is Strata with IQ3_XXS at 262k context; the 125B MoE model spills experts over PCIe, leaving about 10.7 GiB of the 63.9 GiB VRAM free.
The user explains the unused VRAM is headroom: 0.1.40.1 caches 24% fewer resident experts than 0.1.39 (17,515 vs 23,168) while cache hit rate only fell from 99.7% to 98.1%, and decode rose from about 87 tok/s to 104 tok/s. The 104 tok/s run used a 250 W power cap versus 370 W for the earlier runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B running at around 18 tok/s on an M1 Max.
Setup is mtplx with 96k context, running reliably.
The user plans to try an agentic flow over Qwen and asks for advice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next at 97 t/s decode on 2x RTX 3090 with Strata, with about half the experts in system RAM.
Setup is Strata v0.1.40.1 with unsloth UD-Q4_K_XL GGUF, a layer split, speculative decoding, and 121 GB DDR4 on a Ryzen 9 3950X.
The 97 t/s is at a 95% expert hit rate; shrinking the cache to 90% and 81% hit rates gave 84 t/s and 68 t/s. The user estimates +34% decode from eliminating misses on their box and about +75% headroom at a single-24GB-card share, and reports +16% decode from --pcie-frac 0.2 --spec-min-p 0.7 over about 280 runs. The prefetch feature is not implemented yet.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User asks how a MacBook Pro M5 Pro 64GB would compare to their work setup of an RTX 4090 with 128 GB DDR5 for running local LLMs.
They report running Qwen3.8-27B at Q4_K_M under llama.cpp on the RTX 4090 at roughly 80 t/s with a ~127k context window, though they note they are on holiday and cannot verify the figures.
The question is whether to buy the 64GB or 32GB MacBook variant for local LLM use at home.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagentic
User reports Qwen3.8 27B IQ2_M at 80-100 tok/s on an RTX 5080 with 64GB system RAM.
Setup is LM Studio with IQ2_M quant; the model is partially offloaded to system RAM because the 16GB card has about 1.5GB used at idle.
The user finds the model too slow for web-search automation and says it loops on web search until hitting the context limit without retrieving the price.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-UD-Q8_K_L at 16.66 t/s generation and 1046.98 t/s prompt processing on an Intel Arc Pro B70 GPU.
Setup is llama.cpp with the SYCL backend, FP16 build, flash attention off, single GPU, on Windows 11 with a Ryzen 9 5950X host.
With flash attention on the same setup gives 16.63 t/s generation and 572.10 t/s prompt processing; FP32 builds give 16.64 t/s and 335.62 t/s without FA, and 16.60 t/s and 300.36 t/s with FA.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-context
User benchmarks Qwen3.8-27B at 41.4 tok/s decode and 198.2 tok/s prefill at 1K context on an Apple M3 Max 128GB (40-core GPU).
Setup is MTPLX v2.8.2 with a 4-bit dynamic/GS32 MLX model, MTP depth 3, single request, 128-token generations, 3-minute GPU rests between rounds.
MTPLX decode falls to 19.8 tok/s and prefill to 115.5 tok/s at 16K, where oMLX oQ4e reaches 27.0 tok/s decode and 231.7 tok/s prefill; the 8-bit MTPLX Quality build is nearly identical in speed (18.8 tok/s, 121.3 tok/s) with KL 0.00105 versus Speed's 0.0220. ANE prefill on oMLX gains 57.0% at 32K (126.1 to 198.1 tok/s).