Qwen3.8 reports

generation:

157.0 tokens/s

quant:

EXL3 3.5bpw (EXL3)

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 157 tok/s decode on an RX 7900 XTX 24GB with DFlash2 speculative decoding at 2K context. Setup is a ROCm port of ExLlamaV3 with EXL3 3.5bpw weights, Q8 KV cache, and a DFlash2 5.0bpw draft model, served through TabbyAPI. Decode drops to 147 tok/s at 32K and 107 tok/s at 99K; MTP gives 108 tok/s at 2K. Prefill is ~1180-1200 tok/s at 32K. Through the TabbyAPI OpenAI endpoint, code reaches ~186 tok/s and prose ~78 tok/s. The same GPU with llama.cpp and an IQ3_XXS GGUF decoded at 40-53 tok/s.

generation:

34.8 tokens/s

prompt processing (prefill):

501.0 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 Flash-Next at 34.8 tok/s decode on two Tesla P40 24GB cards with a layer split, and 19.6 tok/s on a single P40. Setup is Strata v0.1.39 with IQ2_XS weights at 32,768 context, experimental CUDA 12 engine built locally; the two-card run splits layers across both GPUs (K=24, all 24576 profiled expert pairs resident). Prompt reading was 353 / 501 tok/s at 4K / 16K on both cards and 348 / 388 tok/s on one; figures are medians of 3 runs per cell, with the first run of each configuration slower due to expert-cache warm-up. A 69-task correctness suite ran twice, and two greedy runs of identical prompts gave 49 of 69 identical outputs.

generation:

27.3 tokens/s

quant:

UD-Q3_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 27.33 tok/s decode on an M4 Max 128GB. Setup is llama.cpp (current main, 62acc89) with UD-Q3_K_XL GGUF at 32768 context, -ngl 99, single stream. The MTP head is not exported in any GGUF of this model, so no speculative decoding. On the same box an MLX pack with MTP reached 90.07 tok/s and 45.01 tok/s with MTP off. The GGUF is 90 GB of weights versus 46.9-49.1 GB resident for the MLX pack. Quants are not matched and the MLX pack is an abliterated fine-tune, so this is a throughput comparison only. LM Studio cannot load the model on either its GGUF or MLX runtime.

generation:

20.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next-Heretic Q8_0 at 20.45 tok/s on an M5 Max 128GB. Setup is llama.cpp (closed PR #27739 fork) with Q8_0 and Q4_0 KV cache, -ngl 16, 4096 context, 45.1 GiB wired, MTP draft head enabled. Measured on 12 varied prompts with 24 warm-up queries discarded. The model is 122B total / 10B active MoE, 197 GB across 5 shards. A 262144-token server ran on llama.cpp v0.6.0 with ~32 tok/s prefill and decode falling to ~3.6 tok/s at 10k tokens. Under memory pressure a larger GPU tier wins: -ngl 24 at 10.29 tok/s with 15 GiB held by other work versus -ngl 6 at 5.75 tok/s with 45 GiB held. Contiguous layer placement beats scattered expert placement by 28.6%. MTP acceptance is 51-83%, helping +44% when experts are cached in RAM but costing 16-33% when they stream from SSD. Stock llama.cpp was 11-19% faster than the fork at 4k/64k/256k, and v0.6.0 another ~7% faster. No quality measurement was run for this file.

generation:

158.5 tokens/s

prompt processing (prefill):

1636.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 158.5 tok/s decode on a Radeon AI PRO R9700 in an iPad Pro eGPU enclosure. Setup is LemonSeed Engine with Q4 weights and a Q8 DFlash2 draft, 131k context, speculative decoding on. Baseline decode is 31.2 tok/s and MTP=3 gives 108.7 tok/s; prefill at 4K is 1,636 tok/s and 1,395 tok/s at 32K. On Strix Halo (Radeon 8060S) the same model reaches 64.4 tok/s decode and 517 tok/s prefill at 4K.

generation:

47.6 tokens/s

prompt processing (prefill):

1672.8 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 47.597 t/s generation and 1672.768 t/s prompt processing on an M5 Ultra 256GB Mac Studio. Setup is native mlx_lm.benchmark (mlx-lm 0.31.3 on MLX 0.32.2) with 4-bit MLX weights, 32768 prompt tokens and 128 generated tokens over 5 trials, peak memory 20.143 GB. Figures are medians from a third-party review unit; generation ranged 47.558-47.610 t/s and prompt processing 1671.556-1673.415 t/s. Speed only, no quality evaluation, and no hardware speedup ratio is claimed.

generation:

70.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

W4A16-AutoRound (AutoRound)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~70 t/s generation and ~2800 t/s prefill on 4x Intel Arc Pro B70 GPUs. Setup is vLLM with Intel's W4A16-AutoRound quant, full 262K FP8 context, prefix caching, and concurrency=2. The user notes an alternative setup reached ~100 t/s generation but only ~1400 t/s prefill and 16-bit context, and considers the doubled prefill and context worth the ~30 t/s generation loss.

generation:

33.6 tokens/s

prompt processing (prefill):

1944.0 tokens/s

quant:

4.05bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks a MoE CPU-offload patch for ExLlamaV3 that streams expert weights directly from a page-locked shared arena, removing the host staging memcpy. On an RTX 4090 with a Ryzen 9 7900X, the Qwen3.8 Flash-Next 125B-A6B model at 4.05bpw reaches 1944 tok/s prefill at 4096 context and 33.6 tok/s decode at context 0 on Windows, roughly doubling upstream prefill. Setup is ExLlamaV3 with EXL3 4.05bpw weights, 32768 context, chunk size 4096, and CPU-offloaded MoE experts. The patch is PCIe-bound at 26-30 ms per ~1 GiB layer. Prefill figures are medians of 5 runs with min-max ranges; WSL2 reaches 2044 tok/s at 4096 context. Decode varies more with machine state than with the patch.

generation:

100.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B MXFP4 at over 100 t/s on 2x R9700s, with over 200 t/s for an initial request and around 80 t/s under concurrent requests. Setup is DeepSeek Harness with MXFP4 weights on two R9700 cards, running at 130k context out of a 262k limit. The user is promoting a paid token service built on this setup rather than a personal benchmark.

generation:

38.3 tokens/s

prompt processing (prefill):

857.0 tokens/s

quant:

BALANCED-2.1 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 38.3 t/s generation and 857 t/s prefill on a Strix Halo 128GB. Setup is a ROCm llama.cpp branch with the BALANCED-2.1 GGUF quant (78.3 GB) and MTP speculative decoding, single card, 8K context. The 38.3 t/s is the short-answer figure at 8K; code writing reached 46.7 t/s and long prose 30.1 t/s at the same depth. Prefill fell from 857 t/s at 8K to 301 t/s at 256K, and generation from 38.3 to 25.2 t/s. GPU memory stayed at 68-70 GiB across the full window.

generation:

42.4 tokens/s

prompt processing (prefill):

404.0 tokens/s

quant:

ROCmFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 42.4 t/s writing code on one Strix Halo 128GB, with 404 t/s prompt reading at 8K context. Setup is the paoai-strix-engine strix-v4 Vulkan build with a 15 GB ROCmFP4 GGUF, MTP draft head for the Chat profile, DFlash2 adaptive drafting for Code and Long, and an f16 KV cache for Long. Three serving profiles were measured at 8K, 32K, 64K, 128K and 256K with fresh prompts and prompt cache off; the 256K window was verified on all three, and the Code and Long profiles scored 100 on a coding exam. A 55-task quality bench scored 83.2 for Chat.

generation:

60.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 60+ tok/s decode and 2000+ tok/s prefill on Strix Halo 128GB plus an R9700 32GB. Setup is a modified llama.cpp (llama-halo-hybrid) with Q4_K_XL weights, splitting dense layers and KV cache to the GPU while the APU runs the rest, supporting full 256k context. The user says it beats DGX Spark running the same model, with the Swift-1.5 variant performing slightly better, and recommends Strix 128GB plus R9700 over Gorgon Halo for price and performance.

generation:

21.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 21 t/s on two modded RTX 3080 20GB cards, with MTP raising it to 33 t/s. Setup is llama.cpp with Q8 weights and 131.1k context plus vision, run on the two 3080s; the 3080s are power limited to 220W. The user also ran Gemma 4-26B-A4B on a single RTX 4070 Ti 12GB and found coding output poor, and has just reinstalled the 4070 Ti so has limited data with all three cards.

generation:

95.0 tokens/s

quant:

IQ4_XS-M (GGUF)

kv:

TQ5/TQ4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 95+ t/s generation for Qwen3.8 27B on a single RTX 3090, sustained through 100K generated tokens at 262K context. Setup is the llamAmpere fork of llama.cpp (v0.4) with an IQ4_XS-M GGUF quant and TQ5/TQ4 KV cache, full GPU offload (-ngl 99), flash attention on, temp=1. The figure is a minimum across the run; the user notes ~10% speed gain over the previous version and that the KV quant adds under a third of the KLD of the 8-bit to 4.6-bit weight quant.

prompt processing (prefill):

600.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 600 t/s prefill and 22-26 t/s generation on an RX 9070 16GB. Setup is llama.cpp with IQ3_S GGUF and q4_0 KV cache at 131072 context, using about 15.2GB of VRAM. MTP is not active because it did not improve performance and used more VRAM; generation starts at 26 t/s and degrades to 22 t/s at full context.

generation:

53.0 tokens/s

prompt processing (prefill):

2200.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 53 t/s generation and 2,200 t/s prefill on a Mac Studio M5 Ultra with 256 GB unified memory. Setup is MTPLX 2.12.0 with speculative decoding and 55-65% draft acceptance; quantization and KV cache are unspecified. The 53 t/s figure is at roughly 4K context, with 51 t/s at roughly 16K context; the 2,200 t/s prefill is at a roughly 17K prompt. Resident memory was not measured; the 99 GB figure is the on-disk size only. Measurements are medians of three runs per cell, and the author notes small models served traffic on the same GPU throughout testing.

generation:

28.4 tokens/s

prompt processing (prefill):

314.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next at 28.4 t/s decode and 314 t/s prefill on a Minisforum S1 Max with AMD Ryzen AI Max+ 395 and 128 GB unified memory. Setup is llama.cpp (Vulkan, Unsloth build) with UD-Q4_K_XL weights and MTP speculative decoding at 56% accept rate. The repo also lists community engine comparisons on the same hardware: gufo 34.1 t/s decode and 1,075 t/s prefill, Halogen 39.4/1,191, strixllama 29.7/716, and llama.cpp ROCm 20.5/285. The GGUF checkpoint uses a shared BF16 PLE of about 95 GiB that should stay NVMe-backed via mmap on 128 GB machines.

generation:

166.7 tokens/s

prompt processing (prefill):

2719.9 tokens/s

quant:

GSQ (safetensors)

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 166.7 t/s generation and 2719.9 t/s prefill on an RTX 4080 16GB. Setup is NInfer with GSQ safetensors and rk4v4-e8 KV cache, single card, DFlash2 speculative decoding at 8K context. User also reports MTP3 decode at 151.2 t/s and DFlash2 K=7 decode at 262.3 t/s at 32K, with prefill up to 2720 t/s and 100k context support.

generation:

70.0 tokens/s

quant:

8-bit

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 70+ tok/s generation on an M5 Max 128GB MacBook Pro. Setup is the MLTF engine with 8-bit weights and an INT8 KV cache, using Metal TensorOps W8A8 kernels. Cold prefill is 1,000+ tok/s on a 2Ki prompt, and aggregate generation throughput is 200+ tok/s at concurrency 4. The user chose 8-bit over 4-bit for quality.

generation:

241.7 tokens/s

quant:

2.50bpw (EXL3)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 241.7 tok/s on code and 254.3 tok/s on prose on 2x RTX 5090. Setup is ExLlamaV3 v1.5.0 with TabbyAPI, 2.50 bpw EXL3 quant, 8-bit KV cache, 262k context, MTP speculative decoding on, 2 cards. Aggregate decode at 4 streams is 592 / 595 tok/s; prefill is 10,034 / 9,865 t/s at 86-172k-token prompts; peak decode power 302.8 W / 285.0 W per card.