Qwen3.8 reports

generation:

158.0 tokens/s

prompt processing (prefill):

7265.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 158 tok/s decode and 7,265 tok/s prefill on an RTX 5090 32GB (eGPU via OCuLink Gen4 x4). Setup is NInfer with NVFP4 weights, FP8 KV cache, 240K context, MTP3 speculative decoding at 76% acceptance, and 2 concurrent lanes. Decode rises to 213 tok/s at 32K and 202 tok/s at 128K; prefill is 6,892 tok/s at 32K and 3,904 tok/s at 128K. User compares against llama.cpp (Q5_K_M GGUF, q8_0 KV, 196K context, MTP on) at 114 tok/s decode and 1,545 tok/s prefill at 1K, and notes vLLM at about 70 tok/s decode at short context. Quality was statistically indistinguishable across engines on a 250+ item eval. NInfer does not support json_mode.

generation:

48.0 tokens/s

prompt processing (prefill):

530.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares three Qwen3.8 27B Q4_K_M quants on an RX 7900 XTX: the base unsloth quant, ThinkingCap and Swift, all run with thinking at xhigh. The base quant decodes at ~48 t/s with ~530 t/s prefill, ThinkingCap at ~43 t/s with ~100 t/s prefill (TTFT 6-11s), and Swift at ~32 t/s with ~614 t/s prefill (TTFT ~1s). Total tokens per 4-scenario run fall from ~66k for base to ~49k for ThinkingCap and ~45k for Swift, with quality in the ~84-87 band for all three; the user notes only 2 runs per quant and calls the ThinkingCap prefill oddly low.

generation:

138.0 tokens/s

quant:

GPTQ-int4 (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~138 t/s single-request on an RTX 3090, up from ~124 t/s, using a custom vLLM 0.27.1 stack with DFlash2 speculative drafting. Setup is vLLM with GPTQ-int4 weights, fp8 KV cache, 64k context, and a W4A16 DFlash2 drafter; the 3090 is power limited to 250W. The same stack reaches 942 t/s at 64 concurrent requests. User notes DFlash2 is best for 1-4 concurrent users and MTP wins at 8+ concurrent; quality is unchanged with perplexity 8.09 and GSM8K 96.5%.

generation:

107.0 tokens/s

prompt processing (prefill):

5008.0 tokens/s

quant:

Q4/Q5

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 107 tok/s decode with MTP on mixed prompts and 5,008 tok/s prefill at 2K tokens on one RTX 4090 under native Windows. Setup is NInfer with Q4/Q5 weights, rk4v4-e8 KV cache, 262K context per request, MTP with 3 draft tokens and n-gram chaining. Decode reaches 289 tok/s with MTP + n-gram on edit-style prompts and 211 tok/s with DFlash2 on code; prose with thinking on gives 99 tok/s with DFlash2 versus 85 tok/s with MTP + n-gram. Against llama.cpp on the same card, prefill is 5,008 vs 2,729 tok/s at 2K, a 64K prompt takes 17.2 s vs 29.8 s and a 128K prompt 43.0 s vs 73.6 s. Ternary Bonsai 2 27B reaches 188 tok/s decode on mixed prompts, 532 tok/s with MTP + n-gram on edits and 360 tok/s aggregate with three concurrent requests. The 4090 also drives a 4K monitor at 60 Hz, costing about 15% of decode.

generation:

77.2 tokens/s

quant:

NVFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 77.2 t/s decode after a 1K prompt and 64.7 t/s with 128K resident on a single RTX 5090. Setup is vLLM 0.27.1 with an NVFP4 ModelOpt export, FP8 KV cache, prefix caching, vision and tool calling, at a 262,144-token context; the 19.18 GiB checkpoint leaves about 1.6 GiB free VRAM with KDE and Firefox running. Prefill falls from 7,005 t/s at 8K to 1,578 t/s at 262K, and a 262,000-token prefill took 166 seconds. A shared 36,864-token prefix cut TTFT from 6.437 s to a 0.288 s median, a 22.3x speedup. The 128K and 262K rows are single runs.

generation:

30.1 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Q8_0 at about 30 tok/s decode on 3x AMD Instinct MI50 16GB (gfx906) with 262,144 configured context. Setup is llama.cpp with HIP/ROCm 6.2.4, RCCL, tensor split mode, F16 KV cache, flash attention on, and about 43.7 GiB aggregate VRAM use. At 216k occupied context decode was 18.24 tok/s with 187.75 tok/s prompt throughput. With HIP graphs enabled, a tiny request after a ~236k stress run stayed at 8.74 tok/s until the server was restarted; disabling graphs (GGML_CUDA_DISABLE_GRAPHS=1) cost about 3% at short context but restored ~30 tok/s immediately after the long-context run.

generation:

50.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 50 t/s on an M5 Pro 64GB. Setup is Inco Splash with Unsloth UD-Q4_K_XL GGUF weights, run in an agentic setup. The user credits Splash's optimized kernels, speculative decoding, prefix cache and mixed-weight support, and calls it a breakthrough in local inference on Apple Silicon.

generation:

22.2 tokens/s

prompt processing (prefill):

245.5 tokens/s

quant:

Q4_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks a Qwen3.8-27B Ridge Intel Arc tuned GGUF on an Intel Arc Pro B60 24GB, reporting 22.23 tok/s pure autoregressive decode and 245.54 tok/s prompt prefill. Setup is llama.cpp with Q4_K weights, -ngl 99 and -fa on, on a single 24GB card. With embedded MTP speculative decoding the tuned Ridge build reaches 41.28 tok/s at 93.4% acceptance, and the DAS Lab IQ3_S tuned build reaches 41.88 tok/s at 92.9% acceptance. Stock IQ3_XXS decodes at 8.10 tok/s and stock Unsloth UD-Q4_K_S at 12.97 tok/s. At ~150k-token depth decode falls to 7.91 tok/s with DFlash2 and 7.21 tok/s with native MTP.

generation:

77.6 tokens/s

prompt processing (prefill):

14-37 tokens/s

quant:

BF16 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 77-80 tok/s on a single RTX PRO 6000 Blackwell 96GB. Setup is SGLang with BF16 safetensors, FP8 KV cache, EAGLE speculative decoding, and 262144 context length. The decode figure is a range; EAGLE accept rate was 0.94-1.00 and the user claims about 2.2x speedup over non-speculative decoding. Prefill was 14-37 tok/s on 122-token batches.

generation:

120.0 tokens/s

quant:

W4A16 (safetensors)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 120 t/s on Qwen3.8-27B (Swift fine-tune) with a 150k context window on four RTX 3060 Ti 8GB cards. Setup is vLLM with W4A16 weights, bf16 KV cache, tensor parallel across 4 GPUs, and MTP speculative decoding; cards are power-limited to 110W. Quantizing the KV cache to kv8 extends context to the full 262k but drops speed to around 70 t/s. User also reports running 2 concurrent agents with little slowdown, and previously got about 70 t/s on a Swift-Qwen3.8-27B EXL3 4.00bpw model with MTP and 196k context under ExLlamaV3.

generation:

80-120 tokens/s

quant:

6bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-120 t/s on 6x RTX 3090. Setup is ExLlamaV3 with a 6bpw EXL3 quant from turboderp. The user describes the throughput as a range and praises the engine for CUDA, using it with the Hermes agent controlled from a phone via Matrix.

generation:

113.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 113 t/s on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 and tensor parallelism 2, after moving the second card to CPU lanes with an M.2-to-PCIe riser so PCIe atomics and P2P work. User compares Windows LM Studio (31 t/s on 2 cards, 38 t/s on 1), Vulkan with MTP (39-55 t/s), Linux llama.cpp Q8 tensor split (34.4 t/s, up to ~52 on other prompts), vLLM MXFP4 TP=1 (45 t/s) and vLLM FP8 TP=2 (72 t/s). User settled on Qwen3.8 27B Q8 at around 72+ t/s for daily use.

generation:

40-50 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen3.8 Flash-Next on a Minisforum Strix Halo 128GB box, sustaining 40 to 50 tokens per second on a one-shot meteor game smoke test. Setup uses Vulkan rather than ROCm, with MTP speculative decoding at depth 4 and context set to 128K minimum. The user notes the box cost £3500 and that the DGX Spark is about 20% more expensive with similar 256GB/s memory bandwidth.

generation:

80.0 tokens/s

quant:

W4A16-FP8PLE (W4A16)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80 t/s in real world scenarios on dual RTX 3090s. Setup uses the W4A16-FP8PLE quant with context reduced to 220k and hot experts increased to 88 for max speed. User also reports 2k+ prefill.

quant:

Q4_K_M (GGUF)

User reports running Qwen3.8-27B on a 2x3060 machine with llama.cpp, achieving generation speeds ranging from 43.37 to 49.99 t/s and prompt processing speeds from 45.01 to 198.10 t/s. Setup uses Q4_K_M quantization, 50000 context length, tensor split, flash attention, and speculative decoding with ngram-mod and draft-mtp. User notes the main limitation is short context, which is too short for long agentic sessions.

generation:

20-40 tokens/s

quant:

exl3-4.0bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 20-40 tokens per second on 2x RTX 3060 12GB. Setup is ExLlamaV3 with an exl3-4.0bpw quant at roughly 170k context. The user asks for tips to improve performance.

generation:

60-88 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen 3.8 27B FP8 on a single RTX PRO 6000 Blackwell 96GB with vLLM, shared by a small group of developers for coding agents. Setup uses vLLM with prefix caching and speculative decoding. In its first week the node processed 2B tokens, with two users alone accounting for roughly 950M and 900M tokens. Per-user throughput while sharing the node is 60-88 tok/s. The user reports 0 preemptions, 0 queued requests and 38.6% peak KV cache usage, and notes the node is not yet saturated.

quant:

Q8_0 (GGUF)

User reports running Qwen3.8-27B at 44-46 t/s at the start of context and 28-30 t/s as context fills, on three RTX 3090 GPUs. Setup is self-compiled llama.cpp with Q8_0 GGUF, 262144 context, tensor split 14,24,24, and ngram-mod plus draft-mtp speculative decoding. The 3090s are power-limited to 300W. The user notes the machine is a primary desktop, so 2-3 GB of VRAM is reserved for the OS and applications, leaving roughly 62GB usable and 10-12 GB spare at full context.

generation:

14-22 tokens/s

prompt processing (prefill):

130-200 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 130-200 t/s prefill and 14-22 t/s generation (about 15 t/s on average) on dual RTX 3090s. Setup is llama.cpp with Q4_K_XL GGUF and q8_0 KV cache at 131072 context, 48GB VRAM across two cards, 128GB DDR4 and an old 40-core Xeon in a Proxmox VM with PCIe passthrough and NUMA binding to one physical CPU. Prefill is thought to be CPU-limited; ngl, ncmoe and tensor split were manually adjusted to fit the model without crashing.

generation:

9.5-10 tokens/s

prompt processing (prefill):

12-38 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 9.5-10 t/s decode and 12-38 t/s prefill on dual RTX 5060 Ti 16GB cards with 96GB DDR4 system RAM. Setup is Unsloth Studio (llama-server backend) with UD-Q3_K_XL quant, q4_0 KV cache, 65k context, experts offloaded via --cpu-moe -ngl 99 -b 1024 -ub 512 -fa on, and MTP speculative decoding with an ~87% draft acceptance rate. User asks whether ditching MTP, bypassing mmap with --load-mode none, or building a custom llama.cpp with the LRU expert cache would help on DDR4, or whether the setup is bandwidth-capped.