- generation:
- 25.0 tokens/s
- quant:
- GSQ-RCO
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 25 tok/s on an RX 6900 XT installed in a retired HP DL380p server with 172 GB DDR3 and 2x 10-core CPUs.
Setup is Strata with llama.cpp and an HTTP router, GSQ-RCO quant at 64k context. The GPU is powered by an external desktop PSU; total draw is 250 W idle and about 450 W while inferring.
User also reports Qwen3.8-27B with GSQ-RCO-IQ3_XXS and MTP at 192k context reaching 45 tok/s. The machine is a prototype; user plans a flexible riser and a proper GPU platform, and may add Nvidia P40s in the remaining slots someday.
- generation:
- 35.3 tokens/s
- prompt processing (prefill):
- 528.9 tokens/s
- quant:
- Q8_0 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 27B at 35.28 t/s decode and 528.91 t/s prefill on an M5 Pro 64GB Mac Mini.
Setup is Splash with Q8_0 GGUF weights and 4k input prompts.
User also compares llama.cpp, MLX, and Splash across several quants, and reports an M1 Max 64GB Splash run at 135.96 t/s decode and 915.01 t/s prefill. User settled on Splash with Q8_0 and is happy with the results.
- generation:
- 130.0 tokens/s
- prompt processing (prefill):
- 5000.0 tokens/s
- quant:
- FP8 (FP8)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B (Swift 1.5) at 130 t/s generation and 5000 t/s prefill on dual Radeon AI PRO R9700 32GB cards.
Setup is vllm-radiance with FP8 weights.
User asks whether any other RDNA4-specific engine could beat this, and is interested in running Qwen3.8 Flash-Next, which vllm-radiance does not yet support.
- generation:
- 172.0 tokens/s
- prompt processing (prefill):
- 5905.0 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash Next at 172.0 tok/s decode without speculative decoding on an RTX6000 96GB, at 512 context with non-expert weights downsampled from 16-bit to 8-bit.
Setup is NInfer6000, the user's fork of NInfer, with NVFP4 quants and prefill at 5,905 tok/s on 512 tokens and 13,908 tok/s on 8,192 tokens.
With MTP3 and --lm-head-draft the same setup reaches 274.8 tok/s at 512 and 401.3 tok/s at 8K. Keeping non-experts at 16-bit gives 118.0 tok/s without speculative decoding.
- generation:
- 46.0 tokens/s
- prompt processing (prefill):
- 1400.0 tokens/s
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 1,400 t/s prefill and 46 t/s decode on a Strix Halo 128GB laptop at 70 W.
Setup is Gufo as the engine, with the model running at xhigh effort. The user notes the model ran on an older Halogen version (0.14.0) and Halogen dropped the connection once, so the last part ran on Gufo.
The user compares the local model against Claude Opus 5.5 on a coding task, finding the local model's PR better in tests and edge cases, though Opus was 2 to 10 times faster overall.
- generation:
- 45.0 tokens/s
- prompt processing (prefill):
- 450.0 tokens/s
- quant:
- Q6_K (GGUF)
- kv:
- Q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 45 t/s generation and 450 t/s prompt processing on dual RX 6800 XT GPUs under Linux with ROCm.
Setup is llama.cpp with Q6_K weights and Q8_0 KV cache, using MTP speculative decoding and tensor parallelism across two gfx1030 cards.
User compares Windows 11 Vulkan results: 35 t/s TG and 180 t/s PP with MTP, 20 t/s TG and 360 t/s PP without MTP, and 15 t/s TG and 350 t/s PP with tensor parallelism and no MTP. User notes a Windows Vulkan bug that halves prompt processing and asks about performance ceilings and other engines.
- generation:
- 44.3 tokens/s
- prompt processing (prefill):
- 870.0 tokens/s
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 44.3 t/s generation and 870 t/s prompt processing on an RTX 4060 Ti 16GB, using a custom llama.cpp fork (llamADALL) built for 40-series cards.
Setup uses MTP with 3 draft tokens, chat template and thinking on, power limited to 133 W, over a PCIe 1x to 16x mining riser. Numbers are the average of 2 runs on a chat growing from 8K to 124K tokens.
The fork roughly doubles base llama.cpp speeds; at 124K context it reaches 33.0 t/s generation and 496 t/s prompt versus 16.6 and 193 for base llama.cpp. User notes removing the power limit improves results by 5-8%.
- generation:
- 33.1 tokens/s
- quant:
- 4-bit group-64 (MLX)
- kv:
- 8-bit
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextsummarizationcoding
User benchmarks Qwen3.8-27B at 33.1 tok/s decode on an M1 Max 64GB, comparing Splash, MTPLX M1 fork, MTPLX upstream, oMLX, and TensorFold across 2K to 256K context lengths.
Setup is Splash 1.1.0-m1 with 4-bit group-64 MLX weights and 8-bit KV cache, speculative decoding enabled, single stream. Splash decodes fastest up to 64K (33.1 tok/s at 2K, 24.1 at 64K) and uses the least memory (26.2 GB peak at 128K).
The MTPLX M1 fork had the shortest whole 3-turn conversation at every length (15% shorter than Splash at 128K) but peaked at 43.5 GB. Upstream MTPLX drops to 5.0 tok/s at 128K. The user forked MTPLX and wrote the benchmark.
- generation:
- 130-180 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen3.8 27B at 130-180 t/s on an RTX Pro 6000 Blackwell Max-Q.
Setup is NInfer with NVFP4 weights, DFlash2 and MTP-7 speculative decoding, on Windows 10.
The user says parallel decoding shows hardly any slowdown and concurrent capabilities are on par with vLLM, and notes the backend is new and still missing features such as advanced sampling parameters and broader model support.
- generation:
- 26.2 tokens/s
- quant:
- NVFP4 (NVFP4)
- kv:
- turbo4
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 26.2 tok/s single-stream on an RTX 5090 32 GB with 254 GiB system RAM.
Setup is FreeToken 0.1.3 with NVFP4 routed experts streamed from system RAM (about 120 GB pinned for offloaded experts) and turbo4 KV cache; MTP is off because verification cost more than direct generation.
Aggregate throughput reaches 47.5 tok/s at concurrency 2, 66.9 at 4, 64.6 at 6 and 68.8 at 8, topping out around 65-77 tok/s, limited by PCIe bandwidth and per-step expert fetch volume.
- generation:
- 29.1 tokens/s
- quant:
- oQ3 (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 29.0822 tokens/s median generation on an Apple M3 Studio.
Setup is oMLX 0.6.3rc3 with MLX-VLM 0.6.3 and MLX 0.32.0, oQ3 mixed-precision MLX safetensors (3-bit affine base, 92.505 GB), native MTP enabled, 262,144-token configured context, greedy decoding over three 512-token runs.
With native MTP disabled the same setup reached 26.5352 tokens/s, a 9.60% improvement; MTP drafted 847 tokens and accepted 583 (68.83% acceptance), and both modes produced matching output hashes.
- generation:
- 73.5 tokens/s
- prompt processing (prefill):
- 1453.0 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-contextvision
User reports Qwen3.8 Flash-Next at 73.5 tok/s with MTP speculative decoding on an M5 Max 128GB, versus 43.8 tok/s plain autoregressive.
Setup is MTPLX with a dynamic 4-bit quant (8-bit attention) MLX pack, 262,144-token context, MTP depth ceiling 3, single stream through mtplx serve.
Prompt processing on a 4,061-token prompt is 1,453 tok/s on MTPLX 2.12.0 (786 tok/s on 2.11.3); a 65,502-token prompt reads at 1,094 tok/s with 63.5 tok/s decode. The pack downloads at 115.1 GB including a 32 GB n-gram table streamed from SSD, peaks at 87 GiB while serving, and is recommended for Macs with 128 GB or more.
- generation:
- 25.7 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 25.67 t/s on an M3 Ultra 256GB.
Setup is Rapid-MLX 0.13.2 with mixed 4-bit affine MLX quantization and a 2048-token prefill chunk, at roughly 102.8-103.8GB active unified allocation.
Decode falls to 24.27 / 23.40 / 21.72 t/s at the 2048 / 8192 / 32768 prompt targets, with median TTFT of 2.262 / 9.236 / 44.659 seconds. Each row is the median of three batch-size-one runs with cold prefix cache and 256 requested decode tokens.
- generation:
- 84.1 tokens/s
- prompt processing (prefill):
- 285-325 tokens/s
- quant:
- 4-bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 84.1 and 72.1 tok/s on production-shaped prompts and 87.0 tok/s at 12K context on a Mac Studio M3 Ultra 256GB.
Setup is mlx-serve 26.9.6-dev with a 4-bit MTP pack and native MTP; prefill measured at roughly 285-325 tok/s, and concurrency gave no gain (1.01x at N=4).
The same model class under Inco Splash reached 85.3-85.7 tok/s on short prompts and 65.7 tok/s at 32K, scaling 1.18x at N=2 and 1.27-1.29x at N=4 before going flat, with the engine reporting a maximum batch width of 4. The user notes the two engines' single-stream figures came from different prompts, so the 27B single-stream comparison is left undecided. Also measured: Qwen3.6-35B-A3B with mlx-dspark at accept length 4.5998 and 2.33x, versus MTPLX at accept 2.2294 and about 1.47x; Qwen3.8-Flash-Next with mlx-serve 26.9.6 and --mtp at 76.8 tok/s single stream (+32% over the same build without the flag) and 133.5 tok/s aggregate at N=8.
- generation:
- 101.6 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagenticlong-contextcoding
User benchmarks Qwen3.8-Flash-Next (125B-A6B) on a single RTX PRO 6000 Blackwell 96GB with 64GB host RAM, comparing vLLM, ExLlamaV3 and SGLang. vLLM with NVFP4/FP8 mixed checkpoint, INT4 PLE offload and MTP reaches 101.6 tok/s at 1k context and 98.9 tok/s at 32k, with prefill around 28k tok/s. ExLlamaV3 EXL3 is faster at short single-user chat (168.9 tok/s at 1k) but fails most requests at 32k concurrency. SGLang with a file-backed n-gram table decodes at 37-51 tok/s. The user notes the 27B incumbent stays in production due to larger KV capacity and reliable tool_choice enforcement.
- generation:
- 53.8 tokens/s
- prompt processing (prefill):
- 1293.0 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 53.8 t/s output and 1293 t/s prompt processing on Strix Halo 128GB.
Setup is Strata with UD-IQ4_XS weights at 8K context; the release adds official Strix Halo support, marked experimental and Linux only.
At 64K the run gives 46.6 t/s output and 1370 t/s prompt processing, and at 128K 51.4 t/s output and 1320 t/s prompt processing. The user says it can go up to 1M context length without big speed loss.
- generation:
- 116.0 tokens/s
- prompt processing (prefill):
- 2908.0 tokens/s
- quant:
- IQ3_S (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticvisionlong-context
User reports Qwen3.8-Flash-Next at 116 t/s decode and 2,908 t/s prompt read on two RTX 3090s with NVLink.
Setup is a modified Strata build (v0.1.38 plus 29 commits) with IQ3_S GGUF at 262K context, greedy, median of 3, on a Ryzen 9 3950X with 121 GB RAM. The second card acts as a peer expert tier read over NVLink; the pair holds about 19,500 of 24,576 IQ3_S experts with a 0.98 hit rate on real agent chats.
Decode gains from the second card are modest (+10%) because the verify window is latency-bound on the primary. Sampled with official settings it sits around 102-104 t/s, a 1.5K prompt reads at ~1,670 t/s, and switching chats takes 1 s. One stream at a time; answers are word-for-word identical to the one-card run.