Qwen3.8 reports

generation:

179.7 tokens/s

quant:

W3A4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 179.7 tok/s single-stream decode on a single AMD Radeon AI PRO R9700 32GB. Setup is stock vLLM with a plugin, 3-bit rotated GPTQ weights (W3A4) with FP8 KV cache, 262,144-token context, speculative decoding, and a 569,878-token prefix cache. Aggregate throughput is 488.5 tok/s at 8 concurrent requests. Accuracy: GSM8K 95.22, HumanEval 94.51, MMLU-Pro 60.57. The 3-bit weights are a 9.55 GB add-on; MXFP4 remains available at 156.1 tok/s single-stream.

generation:

53.5 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 53.5 tok/s with a cached prefix on an RTX 5070 Ti 16 GB, up from 17.2 tok/s before tuning. Setup is Strata with IQ3_S at 262K context, KV cache in RAM streaming to a 32K VRAM window, speculative decoding with depth 6 and min-p 0.70, and 13 pool workers on a 14700KF with 96 GB RAM. Cold run was 43 tok/s; the 53.5 figure is with the prefix cached, which the user says is what agent sessions actually do. The user also reports the calibrator's own curve at 42.1 tok/s at --pcie-frac 0.0 down to 20.7 at 0.75.

prompt processing (prefill):

7090.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next UD-Q4_K_XL at up to 7,350 tok/s prefill and ~113 tok/s decode on an IBM AC922 with 4x Tesla V100 16GB SXM2 GPUs. Setup is a forked Strata engine with MTP speculative decoding, 72 GiB of experts page-locked in RAM across both sockets, and GPUs pulling from NVLink 2.0 at ~70 GB/s each. The 252K-token prompt prefilled at 7,090 tok/s in 35 s; follow-up at that depth gave first token after 0.26 s and 60 tok/s. Generation varied by workload: ~113 tok/s peak on JSON, ~100 on code, ~84 on prose. Stock llama.cpp on the same machine produced 130 tok/s prefill and 15 tok/s decode.

generation:

50-70 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 50~70 tg/s at 128k context on an M4 Max 128GB. Setup is oMLX 0.7.0 with an oQ4e MLX quant and MTP speculative decoding enabled; memory use is 75~80 GB at that context length. User says older versions dropped from 60 tg/s to 35 tg/s once context passed 30k, and that the speed now holds as context grows.

generation:

36.7 tokens/s

prompt processing (prefill):

899.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next NVFP4 at 36.7 tok/s one-user decode and 899 tok/s prefill at 32k on a Jetson AGX Thor 128GB. Setup is TensorFold 0.6.0 with int8 KV cache, full 262,144-token window, --parallel 4. Prefill measured at 926 / 899 / 798 tok/s for 8k / 32k / 128k tokens; combined decode at 1 / 2 / 3 / 4 users is 34.8 / 53.7 / 70.3 / 76.7 tok/s; first answer to a 60k-token prompt takes 69.7 s. User compares against vLLM's FP8 build on the same box (38.4 tok/s decode, 2,101 tok/s prefill at 32k) and describes a local Thor prompt path with gated runs r09 and r11 reaching 1,549 tok/s prefill at 32k. 128 GB and 273 GB/s are published Thor specs, not re-measured.

generation:

25.0 tokens/s

prompt processing (prefill):

3557.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B at 24.96 t/s generation and 3557 t/s prompt processing on a Jetson AGX Thor 128GB, using the Mjolnir vLLM image with the FA4 GEMV decode kernel. Setup is vLLM 0.30.0 with NVFP4 weights and an 8-bit KV cache at 8K context, single request. The GEMV kernel is default-on and dispatches only on M=1, head_dim=256, GQA shapes. The GEMV leg wins at c=1 but regresses at c=4 (57.74 t/s at 8K context, −15.4% versus stock vLLM), which the user attributes to an open investigation. Stock vLLM reaches 24.43 t/s and 2468 t/s prompt at the same setting.

generation:

33.1 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B decode throughput collapsing from 33.1 t/s at 68K position to 1.4 t/s at 91K position on an RTX 4080 Super 16GB. Setup is llama.cpp (master 8144f31) with UD-Q3_K_XL GGUF and Q8_0 KV cache, no flash attention, 73728 context, single GPU with zero CPU tensor overrides. Prompt processing stays fast at ~1300 t/s at the same 91K position. The user notes the cliff is positional rather than a memory-spill issue, reproduced across q4_0/q5_0/q8_0 KV quants and with FA on/off, and also under Windows native b10488.

generation:

33.0 tokens/s

prompt processing (prefill):

579.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 32.96 tok/s average decode (24.18 stock) on an NVIDIA L4 24 GB, with 39.4 tok/s on prose and 37.4 tok/s across seven workloads. Setup is llama.cpp (PR #27342 plus a MMVQ crossover patch) with UD-Q4_K_XL GGUF, q4_0 KV cache, 226,048-token context, DFlash 2 Q4_K_M drafter at n-max 7, and GGML_MMVQ_MAX=2; 23.0 of 24.0 GiB resident, ECC off, 72 W cap. Prefill is 579 tok/s at a 7k prompt, 514 at 33k and 387 at 79k; mean accepted length is 3.04. The user notes the card cannot reach 100 tok/s because the 15.75 GiB weight read would need 6.7 accepted tokens per pass, and that a 3090-class card would land near 101 tok/s.

generation:

7-10 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B running at around 7-10 t/s on a CMP 50HX upgraded to 20GB VRAM. Setup is llama.cpp b10434 with UD-Q4_K_XL and Q4_K_M quants; the user says context size, KV-cache quantization and MTP on/off made no difference. The user compares this with Qwen 3.6 27B Q4_K_M with MTP enabled, which gives around 25 t/s on the same hardware.

generation:

50-55 tokens/s

prompt processing (prefill):

850-900 tokens/s

quant:

IQ4_XS (GGUF)

kv:

fp16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next Uncensored at 50-55 t/s decode and 850-900 t/s prefill on a Tesla V100 32GB, with general tasks at 40-45 t/s. Setup is Strata with IQ4_XS and fp16 KV cache at 262K context, single card with 180W power cap. User also ran Qwen3.8-Flash-Next IQ3_S on an RTX 3090 at 60-65 t/s decode and 1200-1300 t/s prefill, and notes the uncensored IQ4_XS variant handles non-English writing better.

generation:

21.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 21 t/s on llama.cpp with IQ3_XXS, running on system RAM plus a Radeon AI PRO R9700 32GB. Setup is llama.cpp with IQ3_XXS, model partially offloaded to system RAM (64GB DDR5). User also reports ~60 t/s with the same quant on Strata, and 35 t/s for Qwen3.8-27B Q6 on the R9700. System RAM usage reaches 96% with Flash-Next loaded, preventing simultaneous ComfyUI workloads.

generation:

154.5 tokens/s

prompt processing (prefill):

4183.0 tokens/s

quant:

mixed 4/8-bit (MLX)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen 3.8 Flash-Next (125B-A6B MoE) at 154.5 tok/s decode and 4,183 tok/s prefill on a single Mac Studio M5 Ultra 256 GB. Setup is mlx-serve 26.9.7-dev with the MLX-Serve mixed 4/8-bit pack (100 GiB), KV cache bf16, MTP speculative decoding, PLD and batched decode, at 262,144 context. Decode holds ~150 tok/s through 66k and is 114.4 tok/s at 256k; prefill stays flat at ~4.1-4.35k tok/s. A 4-stream burst gives 166.8 tok/s aggregate versus 119.1 alone. The user notes long-context recall was not verified because answers never cited the planted value.

generation:

44.5 tokens/s

prompt processing (prefill):

404.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0-q5_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 44.5 tok/s decode and 404 tok/s prefill on an RX 7900 XTX 24GB. Setup is llama.cpp (ROCm host build b11160 for gfx1100) with IQ3_S GGUF weights and q8_0/q5_1 KV cache, running the full 262,144-token window with MTP speculation on. Context gate recalled at 213,570 prompt tokens in 528.7 s. Paired MTP measurements for this launch are spec-off 11.2 tok/s versus 23.3 tok/s at 240K fill. The desktop shares about 1.2 GiB of the card, so a headless box may decode slightly faster.

generation:

31.6 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B Q4_K_M at ~31.65 tok/s on LM Studio, compared with ~5.52 tok/s on Ollama, a 5.7x-6.2x speedup. Setup is LM Studio on a consumer dual-GPU workstation, with an automated suite measuring TTFT, TPS, latency and VRAM footprint across 3 prompt tiers. In an OpenCode agentic test, LM Studio completed structured tool calls in 1m 49s while Ollama failed tool-calling and took 7m 50s.

generation:

80-110 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-110 tok/s on 2x RTX 3090 with 96GB DDR5 system RAM. Setup uses the Strata engine with a Q4_K_XL GGUF quant from bitlamas, at 2500 prompt tokens. The user calls it the biggest thing since Qwen 3.8 27B, offering the speed of Qwen 3.5 35B-A3B with higher intelligence than 3.8 27B.

generation:

15.3 tokens/s

prompt processing (prefill):

513.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B at 15.34 t/s generation and 513.39 t/s prompt processing on a single Intel Arc Pro B60. Setup is llama.cpp build b10452 with Vulkan backend, Q4_K_M GGUF weights and Q8_0 KV cache, full GPU offload with Flash Attention, at 512 tokens of context. A second run on two Arc Pro B60 cards gives 14.99 t/s decode and 507.93 t/s prefill at 512 tokens; dual-card prefill scales to 643.16 t/s at 1k, 718.37 t/s at 2k and 149.30 t/s at 64k, while decode stays near 15 t/s.

generation:

24.0 tokens/s

prompt processing (prefill):

609.5 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 24.00 tok/s decode on dual Intel Arc Pro B60 24GB (48GB total). Setup is llama.cpp build b11100 with SYCL F16 JIT, Q4_K_M GGUF, Q8_0 KV cache, 131072 context, and native embedded Q8_0 MTP draft speculative decoding. Prefill is 609.47 tok/s at 512 tokens, scaling to 925.39 tok/s at 4k and 570.24 tok/s at 128k. Stock non-MTP decode is 16.13 tok/s; live llama-server chat with MTP ranges 20.00-30.29 tok/s.

generation:

67.0 tokens/s

prompt processing (prefill):

2600-2820 tokens/s

quant:

escha 2-bit (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-Escha-W2 at 67.0 tok/s single-stream decode on an RTX 4090. Setup is SGLang with a 2-bit escha quant (2.469 bits/weight, 10.15 GB of weights) at 64k context, CUDA graphs on, prefix caching off. With the model's own MTP head as a speculative draft the same card reaches 129.3 tok/s (1.92x). Peak server throughput is 649 tok/s at 16 streams; prefill is ~2,600-2,820 tok/s and TTFT at a 2k prompt is 0.73 s. The card also ran 87.1 tok/s on an RTX 5090 and 40.7 tok/s on an RTX 3090 with ESCHA_ROUTE=blackwell.

generation:

136.8 tokens/s

prompt processing (prefill):

1581.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 136.8 tok/s single-stream decode (8k context, concurrency 1) on 4x RTX 3090. Setup is SGLang with AWQ-INT4 (W4A16 Marlin) weights and bf16 KV cache, tensor parallel 4, DSpark speculative decoding enabled with a 1.36B draft model, 275,018-token KV pool. Prefill sustains 1,581 tok/s at 8k c1 and 1.3-1.6k tok/s across all context lengths; aggregate decode reaches 169.8 tok/s at 8 concurrent requests. Max context 128k, max 9 concurrent requests (GDN state bound). Also benchmarks 2x 3090 (112.9 tok/s decode at 8k c1) and 1x 3090 without DSpark (43.2 tok/s decode, 8k context limit).

generation:

98.2 tokens/s

prompt processing (prefill):

914.0 tokens/s

quant:

3.00bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 98.2 t/s on prose and 225.1 t/s on code on one RTX 3090. Setup is SGLang with the sglang-exl3 plugin, EXL3 3.00bpw weights, fp8 KV cache, a 131,072-token window, and DFlash2 speculative decoding with a 5.0bpw draft. Prompt processing on a new uncached 32k prompt runs at about 914 tok/s with 35.9 s to first token. Compared with the MTP recipe on the same card, code is +57% (225 vs 143) and prose is the same (98 vs 98), with a 131k window instead of 262k. Max concurrency is 1 because two 32k streams stall each other's decode.