Qwen3.8 reports

generation:

66.6-68.3 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen3.8 Flash-Next IQ3_XXS at 59.4-65.2 tok/s prompt and 66.6-68.3 tok/s generation on 4 BC-250 boards with 100K context. Setup is llama.cpp with Vulkan and RPC over 2.5gb ethernet, using speculative decoding with the model's own MTP draft head. The cluster also runs two instances of Qwen3.6 35B A3B at 145 tok/s generation and 450 tok/s prompt with 256K context. Power draw is 600-700 W for Flash-Next and 800-900 W for the two 35B instances. Boards cost under $100 each.

generation:

29.8 tokens/s

prompt processing (prefill):

1193.1 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 29.80 t/s generation and 1193.06 t/s prefill on two AMD Strix Halo 128GB hosts. Setup is Gufo with two-host RDMA tensor parallelism over USB4 RoCE v2, full Q8_0 GGUF, 262144-token configured context, one session, speculative decoding off. Measured prompt was 2040 tokens with no cached prefix and 128 predicted tokens; the 262144 context is capacity, not a benchmark length. Source-reported third-party measurement, not independently reproduced.

generation:

304.0 tokens/s

prompt processing (prefill):

12500.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 304 t/s decode at 8K context on one RTX 5090, with 221 t/s at 60K and prefill about 12,500 t/s at 60K. Setup is a vLLM main build newer than 0.27.1 with a custom NVFP4 checkpoint, DFlash2 drafter with 6 speculative tokens, fp8 KV cache, max model length 195,000 and GPU memory utilization 0.92. Two concurrent slots get roughly 255-299 t/s end to end. An earlier vLLM 0.27.1 setup with qwen3_5_mtp and 3 speculative tokens reached about 123 t/s at 8K and 90 t/s at 60K, with aggregate throughput rising from 67 t/s at one stream to 579 t/s at 16 streams. NVFP4 has a small quality cost versus GGUF Q4_K_M (q_avg 92.5 vs 93.2; HumanEval 89.6 vs 94.5). The GPU draws roughly 380-520 W under vLLM versus about 120 W with llama.cpp.

generation:

42.7 tokens/s

prompt processing (prefill):

1294.2 tokens/s

quant:

UD-IQ4_XS (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 42.7 t/s generation and 1294.2 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB. Setup is Strata at UD-IQ4_XS with speculative decoding on, one request. tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.

generation:

22.2 tokens/s

prompt processing (prefill):

325.3 tokens/s

quant:

MXFP4 / affine 8-bit mixed (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 22.2 t/s generation and 325.3 t/s prefill on an M2 Ultra 192GB. Setup is AX Engine at MXFP4 / affine 8-bit mixed, speculative decoding on, one request. Medians of 20 trials in defai-digital's AutomatosX benchmark of 2026-10-08. A source-reported measurement, not independently reproduced.

generation:

35.0 tokens/s

prompt processing (prefill):

1701.6 tokens/s

quant:

UD-IQ4_XS (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 35 t/s generation and 1701.6 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB. Setup is UD-IQ4_XS with speculative decoding on, one request. tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.

generation:

36.9 tokens/s

prompt processing (prefill):

1232.9 tokens/s

quant:

Q4 Mix (Q4_K/Q5_1/IQ4_NL/Q8_0) (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next Uncensored at 36.9 t/s generation and 1232.9 t/s prefill on an AMD Strix Halo 128GB. Setup is Gufo with MTP speculative decoding on, Q4 Mix (Q4_K/Q5_1/IQ4_NL/Q8_0), one request. Medians of the three mixed-long requests (10,583-token prompts) in vmlinux's bench-gufo-mtp.json. A source-reported measurement, not independently reproduced.

generation:

41.5 tokens/s

prompt processing (prefill):

621.3 tokens/s

quant:

IQ2_XXS gate/up + Q2_K down; BF16 n-grams on SSD (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 41.5 t/s generation and 621.3 t/s prefill on an M4 Max 64GB. Setup is ds4 with IQ2_XXS gate/up + Q2_K down; BF16 n-grams on SSD, one request, speculative decoding off. First row (2,048-token prompt) of pswai's 64gb-mac-llm-bench, measured 2026-09-28. A source-reported measurement, not independently reproduced.

generation:

21.3 tokens/s

prompt processing (prefill):

143.7 tokens/s

quant:

Q4_K_M (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 21.3 t/s generation and 143.7 t/s prefill on an M4 Max 64GB Mac Studio. Setup is ZINC with Metal backend, GGUF Q4_K_M, 48 prompt tokens and 96 generated tokens. Figures are source-published medians from three CLI invocations after one discarded warmup. No speculative decoding is reported. The source warns that 16 of 20 Metal rows have different prompt-token counts between ZINC and llama.cpp, and this submission makes no matched-workload comparison claim.

generation:

36.0 tokens/s

prompt processing (prefill):

1634.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Swift-1.5 Qwen3.8-Flash-Next IQ4_XS on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S) at 36.0 t/s decode for prose, 43.2 t/s code and 40.9 t/s reasoning, with 1,634 t/s prefill at 52k context. Setup is halogen-flash-server 0.16.2 with IQ4_XS weights at 128k context, single stream, temperature 1.0 and reasoning_effort medium. The same engine with base Flash-Next decoded 41.7/51.3/51.6 t/s and prefilled 1,655 t/s at 52k, while the best llama.cpp Vulkan config (Swift Flash-Next IQ3_XXS with MTP n=3) reached 28.8/44.0/46.5 t/s decode and 275 t/s prefill at 52k. Swift-1.5 finished agentic tasks in 40.4 s each versus 53.3 s for base Flash-Next, scoring 36/36 agentic tasks and 56/58 eval items.

generation:

59.7 tokens/s

prompt processing (prefill):

271-330 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 59.7 tok/s cold (c30) on AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, 64GB unified). Setup is llama.cpp b10435-era with UD-Q4_K_XL GGUF, F16 KV cache, 96k context, draft-mtp + ngram-mod speculation (n-max 12, n-min 24). Warm c30 reaches 148-163 tok/s but is labeled a repetition artifact; real conversational traffic is 11-24 tok/s (prose) and 30-40 tok/s (code). Prefill is 271-330 tok/s by depth. Time-per-task is 7.9-14.3s (median 11.3).

generation:

53.7 tokens/s

prompt processing (prefill):

885.0 tokens/s

quant:

4.0bpw (EXL3)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Jev_Qwen3.8-27B at 53.7 t/s decode and 885 t/s prefill on an RTX 4090. Setup is TabbyAPI with EXL3 4.0 bpw and FP16 KV cache, single GPU, max batch size 1, thinking disabled. Decode stayed near 54 t/s at short context and 48.3 t/s at 32K. A warmed TabbyAPI smoke test generated at 53.52 t/s, while a direct ExLlamaV3 probe measured 40.475 t/s. Observed GPU memory was approximately 16.4-16.7 GiB.

generation:

35.9 tokens/s

quant:

EXL3 4.00 bpw (EXL3)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports AREX-2, a 27B dense multimodal agent model built on Qwen3.8-27B, at 35.94 tok/s decode on an RTX 4090 24 GB. Setup is ExLlamaV3 with an EXL3 4.00 bpw quant and FP16 cache, 8192-token runtime context, batch size 2, ~17 GiB artifact. The figure is a small local smoke benchmark, not a standardized result; a TabbyAPI smoke test on the same card saw about 49.4 tok/s on a very short request. Long-context quality and memory scaling at 32K/64K/262K were not validated.

generation:

9.8 tokens/s

quant:

4bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 9.83 t/s generation on a Mac mini M6 24GB, from a 525-token prompt. Setup is mlx-vlm 0.7.3 with the mlx-community 4-bit checkpoint, one request, speculative decoding off. This is the plain baseline from a single run in the author's MTP comparison. The 268 t/s prefill in the source is the prompt divided by time to first token, so no prompt speed is stored.

generation:

1.8 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 1.76 t/s generation on a Mac mini M1 16GB, streaming experts from the internal SSD. Setup is Slotstream 0.2.17 on MLX Swift with PipeNetwork's MLX 4-bit weights, one request, MTP off, a 512-token context and a 19-token prompt. The model's 105 GB sits on disk while the process peaks near 5 GB of memory. A second run gave the same speed.

generation:

25.1 tokens/s

prompt processing (prefill):

153.3 tokens/s

quant:

4bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 25.11 t/s generation and 153.27 t/s prefill on a Mac Studio M1 Ultra 128GB, with a 512-token prompt and 128 generated tokens. Setup is mlxcel 0.7.0-beta.1 with the mlx-community 4-bit checkpoint, one request, speculative decoding off. The figures are one timed pass from the mlxcel benchmark results. The harness counts the first token in prefill, so the decode figure would read about 24.91 t/s over 127 intervals.

generation:

43.4 tokens/s

quant:

4bit (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 43.38 t/s generation on a Mac Studio M3 Ultra 256GB, the median of three requests with an 8,156-token prompt. Setup is Rapid-MLX 0.13.4 with the rapid-mlx MLX 4-bit MTP checkpoint, one request, with the engine's adaptive MTP on by default for this checkpoint. The source's 330.8 t/s prefill is the prompt divided by time to first token, so no prompt speed is stored. The figures come from the Rapid-MLX benchmark docs, not an independent rerun.

generation:

16.7 tokens/s

prompt processing (prefill):

61.7 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 16.7 t/s steady on a Tesla V100-PCIE-32GB, with prompt processing at 61.7 t/s warm. Setup is Strata v0.1.38 compiled for sm_70 with IQ3_XXS, 131,072 context, KV streamed to RAM, MTP draft (q2_0), and 15,553 of 24,576 experts (63%) cached in VRAM (25.2 GiB). Speed stays flat at 16.7-16.9 t/s through a 31k-token session; MTP acceptance is 97.8% on repetitive output and ~65% on free prose. The user also reports a thinking-budget pitfall where hard prompts can exhaust max_tokens on reasoning and return empty content, fixed by sending reasoning_budget_tokens: 4000 per request. A comparison against Qwen3.8-27B on llama.cpp on the same hardware found the 125B stronger on long-tail knowledge and proof depth, while the 27B was 1.7-2x faster in wall clock and better at fine-grained format obedience.

generation:

65.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 65.07 tok/s on a single Tesla V100-SXM2-32GB at 1K context. Setup is llama.cpp (pinned tag b10793) with Q4_K_M GGUF and MTP speculative decoding enabled, 128K context window, single stream. Decode falls to 62.41 tok/s at 8K, 61.24 at 32K, 52.51 at 64K and 42.22 at 128K. With MTP off on a T2-001 packed-attention build the same card gives 35.22 tok/s at 1K and 23.83 at 128K; stock llama.cpp gives 35.95 at 1K and 16.49 at 128K. Prompt caching cuts multi-turn wall time from 22.94 s to 3.08 s per turn.

generation:

29.3 tokens/s

prompt processing (prefill):

1055.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B NVFP4 at 29.3 tok/s single-stream decode on one Tesla V100-PCIE-32GB, comparing 1Cat-vLLM v1.5.1 against llama.cpp. Setup is 1Cat-vLLM with NVFP4 weights and fp8_e4m3 KV cache at 32K context, tensor parallel 1, 29.9 GB VRAM; llama.cpp ran the same model as UD-Q4_K_M GGUF with q8_0 KV cache at 131072 context, 20.6 GB VRAM. 1Cat is about 10% behind llama.cpp on single-stream decode (32.7 tok/s) but 45% ahead with 4 concurrent requests (110 tok/s aggregate vs 75.7) and reads long prompts 25-42% faster (1055 tok/s at 2.9K down to 847 at 29.4K). A separate Qwen3-8B FP16 run on the same card gave 40.7 tok/s single-stream and 141 tok/s at 4 concurrent for 1Cat, against 49.4 and 155 for llama.cpp.