Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Swift-1.5-Qwen3.8-Flash-Next at about 500 t/s prompt processing and 45 t/s generation on a 64GB Strix Halo box.
Setup is Strata with an IQ2_XS GGUF quant on Linux (NixOS), with a lot of memory left free. IQ3_S was tried but decoding was 10 times slower.
The user calls it an upgrade over gufo and llama.cpp, and has not tried halogen.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B with vision at about 37 tok/s on an RTX 5090 Laptop 24GB with 64GB RAM.
Setup is Q4_K_XL at 100K context, tested with the opencode and pi harnesses.
User also ran a two-laptop cluster of RTX 5080 Laptop 16GB machines: Qwen3.8-27B Q6_K fine-tuned for coding agents at 67 to 77 tok/s, Qwen3.8-27B Q5_K_XL at 53 tok/s, Qwen3-Coder-Next Q3_K_XL at 48 tok/s, and Qwen3.8-Flash-Next IQ4_XS at 19 tok/s. On the single 5090 laptop, Qwen3-Coder-Next Q3_K_XL reached 55 tok/s and Qwen3.8-Flash-Next with vision IQ4_XS at 262K context reached 20 tok/s. User says none of the local models can complete even minor tasks, filling context without output or producing nonsense, and that hosted GPT-6 and Opus 5.5 handle the same tasks without issue.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 Flash-Next 125B MoE decoding at 64 tok/s and prefilling ~1,500 tok/s on a Strix Halo tablet, with the NPU handling search, dup scan, decisions and screening.
Setup is the Halogen engine with 262k context, fully local; the GPU still does the thinking and the NPU changed what tokens got spent on, at ~7% iGPU cost when they overlap.
First token lands in ~0.03s, about 43x faster than a cloud call measured side by side and still 7x with a second agent hammering the server; a 194k session compacted in ~50s versus 166 on the main model with 97% cache hit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Token Rush decoding Qwen3.8-27B at 70 tok/s without speculation at 200k tokens of context on one RTX 5090.
Setup is Token Rush with int4 GPTQ weights at 4.25 bits per weight and a 128-group scale, FP8 KV cache, speculation off for this figure, one stream, the whole model in VRAM.
With DFlash2 speculation the same engine decodes 229 tok/s on essay, 358 on code and 379 on math at short context, and 200-220 tok/s on prose at 200k. The full 256k window fits in 26 GB of VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 4-bit at ~25 tok/s decode on an Apple M3 Ultra with 512 GB unified memory.
Setup is oMLX 0.6.3 serving the mlx-community 4-bit weights, the whole model in unified memory, non-streaming chat completions on localhost with max_tokens 400-512 and default sampling.
Speed was taken as completion tokens over wall time: two short-prompt runs at low reasoning effort gave 24.6 and 25.6 tok/s, one at xhigh effort 24.2 tok/s, and a ~1.5k-token prompt 17.9 tok/s including prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next-OrcaRouter IQ4_XS at a bit over 100 t/s decode on short context on 2x RTX 3090 24GB with Strata.
Setup is Strata on 2x RTX 3090 with IQ4_XS weights, which keeps the useful experts on the GPUs and the rest in 64GB DDR5 system RAM, vision enabled, experimental context extension configured for 512K.
A prior vLLM setup did about 60-70 t/s decode. Context extension runs: 200K prompt at ~2,004 t/s prompt processing and ~73 t/s decode; 400K at ~1,633 t/s and ~46 t/s; 900K at ~1,110 t/s and ~44 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-27B W8A8 on 4x RTX 3090 capped at 220 W with 3,843 t/s prefill at 512 prompt tokens (single request); decode is given only per task, 146.6 t/s narrative and 265.5 t/s code, with no overall figure.
Setup is SGLang v0.5.19 on 4x RTX 3090 with TP=4, W8A8 INT8 weights, bf16 KV cache, DFlash2 speculative decoding with 8 draft tokens, HiCache ratio 1.5 and a 262144 context window.
The c=1 prefill sweep ran 3,843 t/s at 512 tokens up to a 5,058 t/s peak at 4,096, and about 4,000-4,300 t/s at 10K and 3,029 t/s at 90K. All four GPUs were capped to 220 W of the 350 W default (62%), so every figure is power-limited. A concurrency sweep reached 1,811 t/s aggregate code decode at c=16 (113.2 t/s per request).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next 125B on an RTX 5060 Ti 16GB with about 620 tok/s prompt processing at 16K; decode is given only per workload, 26.4 tok/s code, 21.3 agent, 21.8 reasoning and 17.7 at 21K context, with no overall figure.
Setup is the author's QwFN-hybrid engine on one RTX 5060 Ti 16GB, the 76GB model's experts offloaded across VRAM, 32GB DDR5 system RAM and an NVMe SSD.
llama.cpp on the same machine prefilled at about 165 tok/s at 16K. The author replays workloads token by token against llama.cpp, and the current release passes 4/5 parity gates.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports the AtomicChat Qwen3.8-Flash-Next GGUF quant on an M4 Max 128GB Mac Studio with cold-start prefill of about 500 t/s; no decode speed is given.
Setup is llama.cpp (qwen4exp branch) with the AtomicChat GGUF, whose PLE n-gram table is paged from its file via mmap, so the model takes about 65GB of RAM instead of 106GB.
For comparison, on oMLX offloading PLE to SSD slows prefill from 600 t/s to 180 t/s, and an open oMLX PR makes that cold prefill almost 3 times faster.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next Q2 at 86.7 tok/s with MTP on an M5 Max 128GB, using sf-q3-8flash, their stripped-down fork of antirez's ds4 engine.
Setup is the sf-q3-8flash fork of ds4 on Metal with Q2 weights, MTP speculative decoding on, the whole model in unified memory. Stock ds4 ran the same Q2 MTP setup at 75.8 tok/s; on Q4 MTP went from 77.8 to 85.9 tok/s. No plain decode figure is given, only a 9-13% decode gain on Q2 and 2-11% prefill gain on Q4.
Output stays bit-exact against stock ds4. An optional SSD streaming mode for experts on a simulated 48GB machine ran Q2 at 27 tok/s, around 35 tok/s with MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B Q8 at 77 t/s decode at 51K depth and 1,816 t/s prefill over a 51K prompt on four RX 7900 XTX cards after a series of ROCm and llama-server fixes.
Setup is 4x RX 7900 XTX on PCIe 4.0 x16 in a Lenovo P620, Q8 weights, a locally patched llama.cpp with MTP speculative decoding.
Before the fixes the same run gave 57 t/s decode at 51K and 1,454 t/s prefill; code decode went from 54-61 to 96-110 t/s. The patches are not yet published.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports 112 tok/s decode on coding prompts with WHIRL, the engine they wrote, for Swift-1.5 27B MXFP4 on a single Radeon AI PRO R9700, against 61 tok/s for llama.cpp b11214 on the same GGUF.
Setup is one Radeon AI PRO R9700 32GB over USB4 on Windows 11, MXFP4 weights, WHIRL v0.1.3 with speculative decoding, single request.
Prefill reaches 1,757 tok/s at 128K and 970 at 256K, and 4 concurrent users get 182 tok/s in total. On Ornith-1.5-35B-A3B MXFP4 WHIRL decodes at 258 tok/s and prefills over 11,000 tok/s at 8K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User (Yamz Labs) reports Qwen3.8-Flash-Next at 32.7 tok/s decode without speculative decoding and 1,412 tok/s prefill at 4K on a single Strix Halo mini PC with their Kyojin engine.
Setup is a single AMD Ryzen AI Max+ 395 with 128GB unified memory, 95GB EXL3 weights, Kyojin on ExLlamaV3, plain decoding with no speculation.
With speculative decoding decode reaches 44 to 59 tok/s depending on the task (chat ~47, code ~58, copy-heavy edits ~60). Prefill stays near flat at 1,486 tok/s at 32K and 1,367 at 128K. The authors say Halogen 0.16.2 is faster at 39.8 tok/s plain.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 220.84 tok/s decode in their fork's README at a 4K prompt for Qwen3.8 27B NVFP4 on two RTX 2080 Ti 22GB cards, and 209.35 tok/s at 32K.
Setup is 2x RTX 2080 Ti 22GB joined by NVLink with tensor parallel 2, NVFP4 weights, FP8 KV cache, a 256K window and DFlash2 speculative decoding (K=7), on a single request.
The figures use synthetic text inputs with high speculative-hit rates, and the README says real-task throughput may not reach them. 4x Tesla T10 with Qwen3.8 27B FP8 reaches 191.89 and 189.38 tok/s on the same test.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports approximately 50 TPS on Qwen3.8 Flash Next on an RTX 5090 with 96GB DDR5.
Setup is a single RTX 5090 with 96GB DDR5 system RAM, with part of the 125B model in system RAM since it cannot fit in 32GB of VRAM.
The user also gets 200+ TPS on Qwen3.8 27B on the same card. No engine or quant is stated, and the post asks which model sits between the two for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 28.5 t/s decode on a 14,644-token prompt on an RTX 5060 Ti 16GB, with prefill at 878-892 t/s.
Setup is an adaptive KV-streaming llama.cpp fork on one RTX 5060 Ti with GSQ-RCO IQ3_S weights, q8_0 KV cache, a 1536 MiB KV window and 131K context allocated, the whole model on the GPU, no speculative decoding; the host is a Ryzen 9600X with 32 GB single-channel DDR5.
Decode falls as the prompt grows: 25.2 t/s at 35K, 20.6 at 55K, 18.7 at 69K and 12.6 at 94K with the 1536 MiB window, and about 9 t/s near 131K. A smaller 512 MiB window drops faster (8.7 t/s at 94K). Prefill falls from 878-892 to 630-635 t/s. MTP is not usable with the fork; it gained about 5 t/s on stock llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 32.1 tok/s plain decode on an unnamed Apple Silicon Mac with llama.cpp master.
Setup is llama-server on one Mac, -ngl 99, flash attention on, one slot, no speculative decoding; the target model's quant is not stated (Q8_0 is the DFlash2 drafter's quant).
The post benchmarks llama.cpp PR #29869 for Metal speculative decoding. Serial decode is unchanged by the PR at 32.0 tok/s. With DFlash2 speculation the per-prompt figures rise from 30.2 to 110.0 tok/s on code at temperature 0 and from 16.8 to 62.6 on prose. Figures are 64 generated tokens, median of 5 requests, mean of two server runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next prefill at 873 t/s on a 4K prompt on Strix Halo 128GB (Ryzen AI Max+ 395, 96 GB dedicated to the GPU) under native Windows 11.
Setup is a Windows port of the Gufo engine on one Strix Halo, UD-Q4_K_XL weights with a Q8_0 MTP head, MTP speculative decoding with survival-based draft stopping and a pruned Latin/code draft vocabulary, 256K context allocated.
Decode is reported separately for a summary prompt and a code prompt with no overall figure: 45.3 and 39.7 t/s at empty context, 40.9 and 39.7 at 4K, falling to 38.4 and 37.1 at 158-174K. Prefill stays between 871 and 930 t/s up to 174K. A follow-up turn on a 158K conversation reached first token in 0.4 s from the prompt cache, then 38.6 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagentictool-use
User reports Qwen3.8-27B at 82.73 t/s median (79.4-89.6) on an RTX 5070 Ti 16 GB at 32K context.
Setup is the den_llama.cpp fork (golden build2) with IQ3_XXS GGUF weights, MTP speculative decoding at n-max 2, accept rate 0.6703, prefill 706 t/s, np=1.
At 65K context the median is 64.16 t/s (51.11-72.48) with q8_0 K / q4_0 V cache. A separate FA-fix build measured 79.48 t/s median at 65K (+23.9% vs control) but was marked experiment, not golden. Build3 adds NVFP4 KV at 88.6 t/s decode (KLD 0.0151) and a Q4_G64 KV codec (KLD 0.0163). A gemma-4-26B-A4B MoE lane runs 94-117 t/s with partial CPU+GPU expert offload (-ncmoe 4), and 129 t/s with q8_0/q4_0 KV and -ncmoe 2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Bonsai 2 at PQ2_0 (2.13 bpw, 7.21 GB) on one RTX 3090 with the llama-cpp-prism ternary fork, prefilling at 834.8 t/s at 10K.
Setup holds the full 262144 context on the single card at 17.1 GB with q8_0 KV; the quant drops the MTP head, so the run is un-drafted. Decode is given per workload only, 42.76 t/s narrative and 41.01 code, with 141 ms TTFT.
The same harness PR adds an exl3/TabbyAPI engine on the 2x 3090 reference rig, whose CPU-computed MoE slugs decode at 50.5 and 39.3 T/s at 204800 context with Q4 KV.