Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports NPU-assisted prefill on a Strix Halo (Ryzen AI MAX+ 395) box using a fork of Gufo, with the GPU and NPU splitting each projection. Prefill reaches 806 t/s on Qwen3.8-27B, 3,276 t/s on Qwen3.6-35B-A3B and 2,125 t/s on Qwen3.8 Flash-Next.
Decode with speculative decoding (MTP, DFlash2, or an auto mode that picks per request) hits up to about 160 tok/s on code and JSON with the 35B-A3B.
Quality checked against llama.cpp on CPU showed no measurable perplexity change on the 27B or 35B. The GPU fabric clock must be held at max or the GPU and NPU together can hang the machine; Linux only, gfx1151 only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-use
User reports Qwen3.8-Flash-Next FP4-Local at roughly 60-78 tok/s around 40-55k real context on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5.
Setup is Strata 0.1.45 with NVIDIA FP4-Local QAD weights, INT8 KV cache at 200k context, a ~38GB RAM expert tier with adaptive VRAM expert cache, SSD-backed PLE and MTP speculative decoding (spec=6).
Cold 43k prompt prefill was about 1,890 tok/s with up to ~90% GPU expert-cache hit. BFCL-v4 quick test scored 85% overall and 92% on the 50 single-turn tool-selection tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at about 30-50 tok/s on an RX 9070 XT paired with an RTX 3080, with performance dropping noticeably at 64k versus 128k context.
Setup is llama.cpp with Q4_K_M weights and Q8 KV cache, layer split rather than tensor split, MTP enabled, one parallel slot and flash attention on. MTP roughly doubles speed; the other tweaks mainly save VRAM.
The user calls the pairing a bad choice for LLM work, citing the 9070 XT's lower memory bandwidth and weaker ROCm/Vulkan performance, and notes task manager shows only about 60% and 30% GPU usage on the two cards. Monitors must stay on the 9070 XT or performance collapses.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at about 100 t/s on three GPUs with Strata, up from about 70 t/s on a single GPU.
RAM use stayed around 80 GB in both configurations, which the user did not expect.
User asks why adding GPUs did not lower RAM use and whether the model can run with 64 GB of RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-usevision
User reports Qwen3.8-27B QUASAR-QAT NVFP4 decoding at approximately 130-168 tokens/s on 2x RTX 5070 Ti 16GB under WSL2.
Setup is capy-flash v0.6.0 built from source, with tensor parallelism across both GPUs and MTP enabled through the 27b profile; MTP acceptance was approximately 49-83%. Model weights are 17.0 GiB, weight loading took 12.1 seconds and total engine startup 29.7 seconds.
Qwen3.8-Flash-Next NVFP4 fails at startup with a cudaHostAlloc out-of-memory error on a host-mapped expert weight buffer of 70039033856 bytes (about 65.2 GiB); the user asks whether WSL2 can support such an allocation or whether the weights could be mapped in smaller chunks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 178 t/s on 2x RTX 5070 Ti with 96 GB DDR5, using their own capy-flash engine with NVFP4 weights.
Setup keeps experts in VRAM with CPU fallback for misses; the same box gives Strata IQ3_S 195 t/s and Strata NVFP4 120 t/s. Prefill on capy-flash is 2.4k / 4.6k / 4.8k t/s at 4k / 16k / 64k.
A replayed 53-turn coding session growing to 81k tokens took 177 s on capy-flash, 197 s on Strata IQ3_S and 268 s on Strata NVFP4. The post also lists many other engines' figures for Qwen3.8-27B and Flash-Next, and EU prices where 96 GB of DDR5 costs about as much as a GPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at 63 tok/s on an M4 Pro 48GB MacBook Pro with the Kapitsa engine, ahead of mlx-serve at 54 and Ollama at 32.
Setup is Kapitsa with 4-bit MLX weights and speculative decoding (DFlash2), with part of each layer's prefill on the Neural Engine.
The post is an engine release with comparison tables across M4 Pro 48GB, M5 Max 128GB and M1 Pro 16GB; on the M5 Max, Qwen3.8 27B reaches 115 tok/s and Qwen3.6 35B-A3B 272 tok/s, while Flash-Next 180B-A7B at 105 tok/s trails mlx-serve and oMLX by 2-3%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B Q4_K_M across two RTX 3090s in llama.cpp, with a 4096-token prompt and TG128 generation. The layer split reached 1,616 t/s prompt and 40.56 t/s generation, against 1,235 / 39.16 t/s on a single GPU.
Setup is llama.cpp build 10775 with F16 KV cache, -ngl 999, flash attention on, no speculative decoding and a single request; the 16.46 GB file fits on one 24 GB card. Both cards ran at PCIe Gen2 x1 with no P2P.
Tensor split was slower: built-in AllReduce gave 140 t/s prompt and 41.10 t/s generation, NCCL 169 t/s and 31.25 t/s. The user concludes layer split was clearly more practical in both speed and latency on this machine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-27B at about 34 t/s per stream across four concurrent requests on an RTX 3090 Ti, roughly 135-160 t/s total.
Setup is vLLM with 4-bit W4A16 weights fully in VRAM, 69.6k context, four requests batched at once; four 17.9k-token prompts read together gave first token after about 19 s (~3,700 t/s total).
A CPU-only Qwen3.6-35B-A3B Q8 slot adds a fifth stream at ~9 t/s writing and 103 t/s reading on a 1.7k prompt (67 t/s at 12.8k). User compares this against Strata running Qwen3.8-Flash-Next 125B IQ3_S at 106-144 t/s writing and ~2,800 t/s reading, and Maya-M running GLM-5.3-Flash 321B at 18-24 t/s writing and ~500-1,150 t/s reading, both one request at a time. The main model, DeepSeek-V4.1-Flash at 2.77 bpw on 2x DGX Spark, writes ~40-55 t/s and reads ~1,600-2,000 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingvision
User reports Qwen3.8-Flash-Next 125B-A6B running a single-prompt Blender room-building challenge on an RTX 5090 32GB, with engine-verified decode median 121.4 t/s and mean prefill 2,757 t/s.
Setup is the strata-nvfp4 fork of Strata with self-converted NVFP4 experts plus Q8_0 dense weights, int8 KV cache, 262,144 max context, MTP draft head with spec 5, and 63 GiB of experts pinned in 192GB DDR5 system RAM.
The run took 26m 17s over 103 steps and 132,929 output tokens with 179 tool calls and no compactions, peaking at 194k context; TTFT was 0.85s median and 32.9s max, and the vision encoder ran on CPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next NVFP4 at 81-88 tokens per second on a single session on one RTX Pro 6000, without MTP.
Setup follows the primitive-ai NVFP4 documentation; the 51B n-gram table needs 100GB of spare system memory. Prompt processing is GPU compute bound at 12000 tokens per second even with context scaled to 100k.
Throughput scales to 341 tokens per second with 6 concurrent clients. The user calls the model great and is still benchmarking with MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 125B at 8.1 t/s on an M4 Mac mini with 24GB unified memory.
Setup is Strata with GSQ-RCO Q2_0 weights, Metal backend, 8K context configured and a 4GB expert weight cache, streaming expert weights from the SSD. The figure is the median of three 128-token generation runs with thinking disabled, temperature 0 and no MTP, on short prompts; the full 8K context was not tested.
User calls the speed slow for everyday interactive work but usable for patient experiments, and notes frequent memory-pressure warnings with no swap use during the short tests.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at Q2 running on a jailbroken PS5 at about 20 t/s decode and 120 t/s prefill.
Setup is llama.cpp-based homebrew (PS5LM) using the console's roughly 11.4GB of usable unified memory out of 16GB, with 448GB/s of memory bandwidth.
User says memory utilization often reaches only 40-60% of the advertised bandwidth and that prefill is the main frustration; a 30B-A3B MoE was too slow to be practical, while tiny ~0.8B models can exceed 200 t/s but loop endlessly.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next on two CMP 170HX cards, with aggregate decode of 226 tok/s for code, 198 for JSON and 157 for prose at one concurrent request.
Setup is a patched vLLM with the W4A16 AutoRound checkpoint, pipeline parallelism and native MTP speculative decoding; the n-gram table is memory-mapped from NVMe and everything else fits on the two GPUs, which run over PCIe Gen2 x4.
Aggregate throughput rises to 840 tok/s for code at eight concurrent requests, where per-request decode was 116 tok/s with 231 ms to first token. Prefill is 6,300-7,100 tok/s on 4k-64k prompts and about 5,900 tok/s at 128k; prose decode falls from 158 tok/s at 4k context to 142 tok/s at 128k. HumanEval scored 157/164 and GSM8K 1,271/1,319, with a full 262,144-token window.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8-Flash-Next at around 40 tokens/s on a single RX 9070 XT 16 GB, with experts partly held in 64 GB of system RAM and read from NVMe when not cached.
Setup is Strata 0.1.44.1 with Unsloth UD-IQ4_XS weights, INT8 KV cache and a 292,000-token context using YaRN; the RAM expert budget is 48 GiB and the engine memory limit 57 GiB.
Prompt processing is about 5,364 tokens/s but may reuse cached context, so it is not a cold-prefill figure. Thinking runs about 36.5 tokens/s and some runs reached about 51 tokens/s. A 49 GiB expert budget was OOM-killed on a roughly 161,000-token conversation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next at 195 t/s decode on two RTX 5070 Ti cards with 96 GB of system RAM, using Strata with IQ3_S weights and experts offloaded to RAM.
Prefill on the same box is 2.2k t/s at 3.5k tokens, 3.8k at 15k and 5.2k at 55k.
On the same hardware capy-flash with NVFP4 and tensor parallel reaches 171-183 t/s decode, and Strata NVFP4 reaches 120 t/s. The user reports Strata IQ3_S beats NVFP4 engines on short decode at about twice the KL divergence (0.144 for IQ3_S vs 0.070 for NVFP4).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 35-50 tok/s on an RX 9070 16GB, used for coding help.
Setup is Unsloth Studio with UD-IQ3_XXS GGUF and q4_0 KV cache, micro-batch 256, context set to 128000, taking 15.50 GiB of VRAM.
The user says the model is stuck at very high thinking and asks about offloading to 32GB DDR5 or SSD, noting 27B spilled outside the GPU is horribly slow and 30B-A3B Q3/IQ3 variants run fast but seem stupid.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next IQ3_S (83.6 GB) decoding at ~15 tok/s (~67 ms/token) in steady state on an RTX 5060 Ti 16 GB with 15.38 GiB RAM and a Gen3-class NVMe.
Setup is a patched llama.cpp (Sabah-MoE-Accelerator) that streams the expert bank from disk into a 9.8 GB VRAM slab with unbuffered I/O; the 28.8 GB per-layer token embedding is left on CPU via -ot, peak VRAM is 15.36 of 16.31 GB, and host RAM stays flat at 4.78 GiB.
Time to first token is 11.2 s for a 53-token prompt (12.68 GB cold fill at 1.11 GB/s), so a 256-token answer is 7.1 tok/s end to end and 10.4 tok/s over the decodes alone; warm-up runs 79-204 ms/token. The author notes speculative decoding and larger batches were 2.8-7.7x slower on this rig, and a second GTX 1080 was measured as no faster than disk.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-uselong-context
User benchmarks Qwen3.8-Flash-Next served four ways on one DGX Spark (GB10, 121 GB unified memory), reporting 45.8 t/s decode and 1,717 t/s prefill for their own fast recipe.
Setup is vLLM with NVIDIA's NVFP4 FP4-Local checkpoint, FP8 KV cache and MTP 3, at a 262,144-token window with thinking off and temperature 0.
The other recipes reach 54.3 t/s (hibrid48) and 51.6 t/s (TensorFold) decode; quality mode drops to 44.4 t/s and 1,362 t/s prefill. Under concurrent load hibrid48 delivers 82 t/s with 16 requests of 8k against 51 for the user's recipe. TensorFold returns HTTP 400 on response_format and scores 6/12 on structured output versus 10/12 for the vLLM recipes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 50.3 t/s generation and 1084 t/s prompt processing.
Source evidence (verbatim):
Source: https://omlx.ai/benchmarks/performance/ogokptif
# Qwen3.8-27B-oQ5e-mtp on M5 Max (40c) - oMLX Benchmark
Hardware: Apple M5 Max (40-core GPU), 128 GB unified memory. Engine: oMLX v0.7.1.dev1. Model: Qwen3.8-27B-oQ5e-mtp, 5-bit. Multi-token prediction (MTP) is enabled with adaptive max depth 3, so this is a speculative decoding run; DFlash and SpecPrefill are disabled. Single request. At 8k context: 1,084 tokens/s prompt processing and 50.3 tokens/s token generation. At 4k context: 1,137 prompt and 58.4 generation; at 1k: 895.8 prompt and 54.6 generation; at 16k: 882.8 prompt and 75.5 generation.