Qwen3.8 reports

generation:

14.0 tokens/s

prompt processing (prefill):

440.0 tokens/s

quant:

FP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B FP4 at 14.0 t/s bare decode on an AMD Strix Halo (Radeon 8060S), with prefill at 440 t/s on pp512. Setup is a llama.cpp fork with a Vulkan backend on stock Mesa RADV 26.0.8, greedy, 300 tokens, with no speculative decoding. With an FP4 DFlash2 sidecar and adaptive drafting (n_min 3, n_max 7) the same model reaches 65.6 t/s on structured output and 26.1 on prose, on a short high-power burst (79 °C, 115 W). On the everyday power profile that configuration reads 55.0 t/s on structured output.

generation:

48.0 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 48 tok/s generation on two AMD Instinct MI50 16GB cards, up from 19.4 tok/s with stock llama.cpp. Setup is patched llama.cpp (b11235) on ROCm 7.14 with Q4_0 weights, tensor split plus MTP speculative decoding with 3 draft tokens, 85 W per card. The 48 tok/s is the patched result; the same setup without the gfx906 kernel patches gave 40-44 tok/s, and a vLLM gfx906 fork with GPTQ-Int4 and TP=2 reached 34.7 tok/s.

generation:

53.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8 27B base, Swift 1.5, and Bonsai 2 on a 16GB RTX 5070 Ti. Swift 1.5 IQ3_XXS at 112K context is judged the best all-round option, nearly matching base on coding and scoring higher on math. Bonsai 2 generated about 1.9x faster (99.6 t/s vs roughly 53 t/s) and supported 256K context but solved fewer coding problems. Base and Swift passed the same 2/4 Next.js tasks; Bonsai passed none.

generation:

35.0 tokens/s

quant:

Q5

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~35 tok/s on an RTX 3090 with 64K context and Q5 quant, used as an always-on personal assistant via Vercel's eve framework. Setup is 64K context, Q5 quant, xhigh reasoning, single RTX 3090. User notes quality is great but speed feels slow for live chat; tried Gemma MoE at 120 tok/s but found it unreliable. Plans to test Q4 and lower reasoning, and recently learned reasoning can be disabled for compaction.

generation:

210.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B NVFP4 at 210 tok/s single-stream on one RTX PRO 6000 Blackwell, using SGLang with the DFlash2 drafter. Setup is SGLang v0.5.20 with a 262,144-token window and 4 slots; the same model without a drafter ran 75 tok/s, and vLLM with DFlash2 ran 160 tok/s. The 27B matrix ran at a 400W power cap. User also reports 607 tok/s aggregate across 4 users, 97s full-window prefill, 73.3% BFCL core, 700/1000 arena score, and 24/40 CAPTCHA puzzles. Flash-Next and an uncensored 27B fine-tune were tested alongside.

generation:

3.8-4 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a decode throughput cliff on a Mac mini M4 Pro 48GB running Qwen3.8-Flash-Next 4-bit via Slotstream. Below 32,768 active tokens decode is about 11 tok/s; above it drops to 3.8-4.0 tok/s. The transition is sharp: 32,320 prompt tokens gives 9.49 tok/s, while 32,770 gives 3.84 tok/s. Setup uses a 26 GB expert pool, 65,536 max context, and MTP off. The user notes MTP on this machine is slower (about 3.7 tok/s vs 12 tok/s off) and disabled it for these tests.

generation:

43.9 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 43.9 tok/s median single-stream on one DGX Spark (GB10, 128 GB unified memory). Setup is vLLM nightly 8a728663 with the official NVIDIA NVFP4 checkpoint, FP8 KV cache, MTP3 speculative decoding, 6 sequences, 4096-token prefill chunk, 262,144-token context, gpu-memory-utilization 0.80; the 47.68 GiB FP8 n-gram table stays on NVMe and is read on demand, giving a KV pool of 1,027,392 tokens. Under load the same boot gives 33.4 tok/s per stream at x2 (43.9 aggregate), 26.4 at x4 (47.1), 21.7 at x6 (68.8). Cold prefill is 1,269 tok/s at 7K, 1,757 at 28K and 1,760 at 113K. The user also compares TP2 and TP4 lanes, where TP2 SPEED reaches 53.7 tok/s median single-stream and TP4 CONTEXT 40.5.

generation:

41.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B in NVFP4 on a single NVIDIA DGX Spark (GB10, 128 GB unified memory), reporting a greedy median of 41.0 tok/s with SGLang + DSpark speculative decoding. Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 max context, and DSpark block-drafter speculation; vLLM + MTP is compared on the same harness. Per-workload SGLang figures range from 37.4-41.8 tok/s on code, 40.1-44.5 on reasoning, 40.3-52.9 on math, and 18.0-19.6 on free prose. vLLM single-stream reaches 20.0 tok/s on random 512/512 and 18.0 on ShareGPT; both engines converge to roughly 60 tok/s aggregate at concurrency 4. SGLang serves 4x the context (262K vs 65K) and has lower TTFT and TPOT.

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

User benchmarks Qwen3.8-27B dense at 38.6 tok/s code generation on a single NVIDIA DGX Spark (GB10, 128 GB unified memory). Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 context and DFlash2 speculative decoding with 8 draft tokens; the same machine also ran vLLM NVFP4 + MTP k3 at 24.5 tok/s and Ollama Q4_K_M at 10.3 tok/s. User reports speculative decoding degrades with context (accept rate 0.36-0.45 short vs 0.10-0.16 long reasoning, decode 27-33 vs 13-16 tok/s), that FP8 is 3x slower than Q4 on dense models (7.8 vs 23 tok/s), and that chat speed is only 15.9 tok/s versus 38.6 on code.

generation:

37.5 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B fine-tune Swift-1.5 at 37.5 t/s decode without speculation on a Radeon AI PRO R9700 32GB. Setup is WHIRL with MXFP4 GGUF weights, one GPU, Windows 11, Adrenalin 26.8.1. With MTP and n-gram speculation the same model reaches 107.5 t/s on Chinese coding prompts and 328 t/s on file-edit prompts; prefill is 3,278 t/s at 8k and 2,595 t/s at 32k. The server serves 199 t/s aggregate across 4 concurrent users. User notes the R9700 is connected as a USB4 eGPU, which slows model loading and cache restores but not prefill or decode. Plain decode on the dense models is only 1.1-1.18x faster than llama.cpp; most of the decode gap comes from MTP and n-gram speculation.

generation:

11.5 tokens/s

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 177B running at 11-15 tok/s on a single RTX 5070 12GB with 32GB DDR4-2400 RAM and a Ryzen 5 5600GT on Windows. Setup is llama.cpp with UD-IQ3_XXS quant and expert streaming from system RAM, using one file handle per worker and a page-locked hot-expert tier. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. Normal conversations run 14-15 tok/s, and a long coding prompt generated 4,892 tokens at 10.15 tok/s. A video shows around 16 minutes for 10k tokens at 10.41 tok/s.

generation:

66.5 tokens/s

prompt processing (prefill):

790.9 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Flash Next at 66.55 tok/s decode on a Ryzen AI Max+ 395 / 8060S Strix Halo 128GB under WSL2 on Windows. Setup uses a 260k-token context; prefill measured at 790.92 tok/s in a separate shorter-output run. Two full 260k sessions ran at 25.45 tok/s each, 50.90 combined, with no aggregate speedup over solo. Windows idle usage is 9-12 GB including Codex; the two-session run left at least 18.86 GB available. Three full contexts froze the PC.

generation:

41.0 tokens/s

prompt processing (prefill):

1167-1191 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next (125B-A6B) at 41.0 tok/s decode on short context and 1183 t/s prefill on a 95.6K-token prompt, on a Ryzen AI Max+ 395 with 128 GB (Strix Halo). Setup is a patched llama.cpp HIP runtime with Unsloth UD-IQ4_XS weights (93.7 GB), MTP speculative decoding with three draft tokens, and optional Q8_0 K/V cache; the GPU carve is set to 96 GB. Decode falls to 32.6 tok/s at 86K context; three and four concurrent conversations reach 55.7 and 60.4 tok/s summed. Prefill is given as a three-run range of 1167-1191 t/s.

generation:

20.0 tokens/s

quant:

GSQ-RCO (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 20 t/s generation on a Strix Halo 128GB laptop with 64GB RAM. Setup is llama.cpp via lemonade with Vulkan on Windows, GSQ-RCO GGUF quant, 16k context, Q4 KV cache, and lazy load enabled. The model uses about 30GB of GPU memory but also consumes roughly 20GB of CPU memory, which the user suspects is the n-gram table not staying on SSD. The user notes the model loaded and worked rather well, but a default prompt ran out of the 16k context due to extended thinking.

generation:

29-46.7 tokens/s

prompt processing (prefill):

1010-1178 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports testing an unpublished engine on Qwen3.8 Flash-Next locally, with reading at 1,010–1,178 t/s and writing at 29–46.7 t/s. The model is a GGUF build (PaoAI STRIX BALANCED 2). The engine is not yet published and is still in testing. The user says the results are promising and that it took 2 turns to build a Pocket Map.

generation:

43.0 tokens/s

prompt processing (prefill):

656.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 125B at 43.0 tok/s decode and 656 tok/s prefill on a single Ryzen AI Max+ 395 (Strix Halo, 128 GB unified memory). Setup is a llama.cpp Vulkan fork with 57 patches, Unsloth UD-Q4_K_XL (111 GB), 8k context, MTP speculative decoding, single request, no CPU offload. At 32k context decode is 40.1 tok/s and prefill 585 tok/s. Inside a tool call decode is about 55 tok/s (was 25), first token after a short tool result 0.4 s (was 1.5 s), and the agent loop over 12 shell commands runs at 16.2 tokens per wall-clock second (was 11.1). KL divergence against full-precision reference logits is 0.116 with 92.2% same top-1 token. The user notes the runtime is reproducible bit for bit and that a PaoAI BALANCED-2.1 file ran 6-9% faster but with about 5x the KL divergence (0.589).

generation:

46.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next (125B-A6B, Unsloth UD-IQ4_XS) at 46 tok/s short and 42 tok/s at 86K context with MTP on a single conversation, on a Ryzen AI Max+ 395 (Strix Halo 128GB) under Windows. Setup is Rulith Inference 0.4 (formerly Strix Llama) with MTP speculative decoding and 2-token drafts. Prefill is 1,237 tok/s over a 95.6K-token prompt. With MTP off, 8 conversations of ~40K tokens each reach 91 tok/s combined; short prompts with typical sampling reach ~100 tok/s. The user runs it as a local worker for coding agents, seeing 30+ tok/s with one agent, 50+ combined with two, and 60+ with three. A caveat is that the local model guesses more (IDs, file paths, field names), so output goes through review.

generation:

22.6 tokens/s

prompt processing (prefill):

63.6 tokens/s

quant:

Q5_K_M (GGUF)

kv:

f16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 22.65 t/s generation and 63.57 t/s prefill on a Strix Halo 395 with 64GB. Setup is llama.cpp (Vulkan build) with a Q5_K_M GGUF and f16 KV cache, 102400 context, MTP speculative decoding with draft acceptance 0.68425. User asks for advice on the best Windows-based setup to reach 20+ tok/s on this model.

generation:

30.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at around 30 t/s on a Bosgame M5 with Ryzen AI Max+ 395 and 128 GB, used for coding. Setup is a Q4 quant; the user also runs Gemma4 26B A4B at around 90 t/s for other tasks. The user asks whether switching to Qwen3.8-Flash-Next is worth it, noting a 4-bit quant of it is around 90 GB and would require dropping the second model.

generation:

33.6 tokens/s

prompt processing (prefill):

1343.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 Flash-Next on a ROG Flow Z13 2025 with Strix Halo 128GB, comparing Halogen, Gufo and Rulith at 60W and 93W. Gufo 0.5.0 with UD-Q4_K_XL and MTP Q8_0 draft 7 reaches 1343 tok/s prefill and 33.6 tok/s decode at 200K context at 93W, and 1156 tok/s prefill with 33.0 tok/s decode at 60W. Halogen 0.16.0 is fastest for prefill, hitting 1694 tok/s at 200K and 41.16 tok/s decode at 93W, while Rulith on Windows is fastest at short context with 54.7 tok/s decode at 1K but falls to 32.55 tok/s at 200K. The user notes the comparison is not apples-to-apples because model formats and MTP setups differ between backends.