Qwen3.8 reports

generation:

97.9 tokens/s

prompt processing (prefill):

2364.0 tokens/s

quant:

Q6_K (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 97.9 t/s decode and 2,364 t/s prefill on an RTX 5090 at 61,136 tokens of context. Setup is llama.cpp with the Unsloth UD-Q6_K GGUF, flash-attn on, q8_0 KV cache, batch 2048, ubatch 1024, and MTP speculative decoding with draft 3. A second run at 97,374 tokens averaged 86.0 t/s decode and 1,929 t/s prefill. The user also tested Qwen3.8-Flash-Next Q4 across both GPUs and Qwen3.8 27B Q8 split across both cards at 262k context.

generation:

87.8 tokens/s

quant:

4bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a Swift-Qwen3.8-27B 4bit model running on the Splash engine on an M5 Max 128GB, with decode of 79–129 tok/s. At 64k context the model reaches 87.8 tok/s versus 36.6 tok/s for the same weights in oMLX (oQ4e-mtp, mixed 4/5-bit), and the gap widens with context. The user notes Splash's /v1 defaults to greedy while Swift wants temp 1.0 / top_k 20 / top_p 0.95, and that the format cannot carry generation_config.json.

generation:

18.0 tokens/s

prompt processing (prefill):

141.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

q8_0

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 18 t/s decode and 141 t/s prefill on a 1.8k-token prompt, running on an AMD Radeon RX 7600 XT 16 GB at 163,840 context. Setup is llama.cpp llama-server build 10480 with the unsloth Qwen3.8-27B-UD-IQ3_XXS GGUF (10.2 GiB), q8_0 KV cache, flash attention on, and Vulkan (RADV) backend. The model is a hybrid architecture where only 16 of 64 layers keep a full KV cache, so KV is about 5.3 GiB at this context. With MTP on the same model runs about 24 t/s at 98k context and 39 t/s at 124k. The user notes 160k is a VRAM-math ceiling, not a quality claim, and that decode drops to about 6 t/s if the GPU spills to system RAM.

generation:

23.0 tokens/s

prompt processing (prefill):

2248.6 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 23.03 t/s generation and 2248.6 t/s prompt processing on a single RTX 4090 with 32 GB RAM. Setup is llama.cpp via llama-swap with the unsloth UD Q4_K_XL GGUF, a Q8 KV cache, 131072 context, and a BF16 mmproj for multimodal input; cache reuse is disabled because of the multimodal model. The user ran a three-day coding session in VS Code Copilot and hit a failure where the model emitted never-ending slashes; they ask what could cause it.

generation:

135.2 tokens/s

prompt processing (prefill):

1402.0 tokens/s

quant:

W4A16 (AutoRound)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 1,402 prompt tok/s and 135.2 output tok/s on 2x RTX 3090 with 128 GB system memory. Setup uses the Intel AutoRound W4A16 target with FP8 PLE/ngram tensors. Prefill and decode use different test shapes and did not come from the same request. A 262,016-token prompt with 128 output tokens prefilled at 1,275.6 tok/s and decoded at 54.5 tok/s. Warm decode over 4,096 output tokens ranged 127.1-134.0 tok/s. Maximum tested sequence was 262,144 tokens.

generation:

29.0 tokens/s

quant:

IQ3-XXS (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~29 t/s on an RTX 3060 12GB at around full context, with 34–40 t/s at lower context. Setup is llama.cpp on CachyOS/Arch Linux with MTP/speculative decoding, comparing the ISTA-DASLab GSQ-RCO-IQ3-XXS quant (~10.4GB, ~2.5 BPW) against the ByteShape IQ3-XXS quant (~500MB smaller, 2.88 BPW). The ByteShape quant performed worse in practice: the GSQ version generated a 3D voxel diorama in one shot under 55k tokens, while ByteShape took roughly three shots and still hadn't completed after over 98K tokens.

generation:

132.0 tokens/s

prompt processing (prefill):

4000.0 tokens/s

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B NVFP4 on dual RTX 5060 Ti 16GB (32GB total VRAM) at 132 t/s decode for structured output and 72 t/s for prose, with ~4,000 tok/s prefill. Setup is stock vLLM 0.30.0 on headless Ubuntu with FP8 KV cache, full 262,144-token context, and MTP speculative decoding. GPUs are power limited to 150W with no throughput cost. Video input is disabled; the user notes this runs without an RTX 5090.

generation:

27.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~27 t/s on a 64GB Mac. The model weighs 95.5 GiB and runs via llama.cpp with a checkpoint and fork. No context length, quant, or KV cache details are given.

generation:

80.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 80+ t/s generation and 1200+ t/s prompt processing on an RTX 5080 with 16GB VRAM, using CUDA unified memory to fit the model. Setup is llama.cpp with the Unsloth UD Q3_K_XL GGUF, 160000 context, q8_0 KV cache, flash attention, and MTP speculative decoding with ngram-mod draft. User asks whether Q3 quants are good enough for harness coding with opencode, pi, and qwen-cli, and notes this is the only quant that fits the whole model in VRAM.

generation:

27.9 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B MoE at 27.9 t/s token-weighted decode on an RTX PRO 4500 Blackwell 32GB with 64GB system RAM. Setup is a llama.cpp fork with UD-Q3_K_XL GGUF, 240,000 context, F16 KV cache, 16 of 48 routed-expert layers on GPU and 32 on CPU, disk-backed PLE, and MTP disabled. Real prompt chunks prefill around 150 t/s with a peak of 201. Decode stayed flat from 116K to 148K context (27.7, 28.5, 27.5 t/s across the three bands). The user notes the sparse-QSA CUDA patch is experimental and unreviewed, and that these are warm-cache workload numbers rather than a controlled benchmark.

generation:

70.0 tokens/s

quant:

INT4

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at over 70 t/s single-stream decode on 2x RTX 3090, with close to 200 t/s decode at 3 to 4 concurrency and over 10k t/s prefill. Setup is stock vLLM v0.29 on Linux serving the RedHatAI INT4 quant with an FP8 KV cache, MTP speculative decoding with 3 tokens, tensor parallel size 2, and the full 262k context. The user notes the setup is standard and that the RedHat quant is overlooked for Ampere hardware.

generation:

48.0 tokens/s

quant:

NVFP4 (FTW)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 48.0 t/s (median 52.4) on 2x RTX 3090 with tensor parallel 2. Setup is a fork of FreeToken with the NVFP4 model converted to FTW, 512 experts offloaded, max prefill length 8192, greedy decoding and MTP off. With MTP on, greedy dropped to 21.5 t/s (median 22.4, accept rate ~56%), so MTP is a net loss on this config; under matched eager conditions it was ~1.5x plain (17.2 vs 11.4).

generation:

71.6 tokens/s

prompt processing (prefill):

1380.6 tokens/s

quant:

Q3G64_F16S/Q4G64_F16S/Q5G64_F16S (ninfer)

kv:

Q4 group64

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 71.57 t/s decode on a single RTX 5080 16GB at 131,072 context. Setup is NInfer v1.3 with mixed Q3/Q4/Q5 groupwise quantization (~3.95 BPW), Q4 group64 KV cache, MTP-3 speculative decoding, and Vision enabled. The 118,001-token prompt prefilled at 1380.61 t/s with 44.74% MTP acceptance. A short 512-token request averaged 84.7 t/s decode, and a video test reached 97.1 t/s decode with 100% MTP acceptance.

generation:

108.0 tokens/s

prompt processing (prefill):

2057.0 tokens/s

quant:

oQ4e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User shares a MacStories review of the M5 Ultra Mac Studio 256GB and a DeepSeek V4.1 Flash comparison against 2x DGX Spark. Qwen3.8-Flash-Next runs at 108 tok/s on a short prompt and 143 tok/s for code generation on the Mac in MLX oQ4e, versus 42.6 and 39.0 tok/s on 2x DGX Spark in NVFP4 TP2. Prefill is 2,057-2,887 tok/s with TTFT of 24.4 s at 64K, 50.0 s at 128K, and 104 s at 256K. Decode decays from 90.7 tok/s at 4K to 74.7 tok/s at 256K, and combined throughput rises from 66.2 tok/s at 1 stream to 81.5 tok/s at 3 streams. The comparison is not a controlled A/B: the Mac runs oQ4e on MLX/oMLX while the Sparks run NVFP4/FP8 on vLLM/SGLang.

generation:

68.6 tokens/s

quant:

PXQ4 (PXQ)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 68.6 t/s on 2× V100 16GB with the PXA engine's new tensor split, up from 51.0 t/s with the layer split it replaces. Setup is PXA v2026.09.20 with the PXQ4 quant format on PCIe x4 risers; the same run hit 107.4 t/s on repetitive text and 43.6 t/s on a ~15k-token prompt. A head-to-head against stock llama.cpp on the same cards gives 80.2 vs 58.9 t/s on prose, 139.8 vs 80.5 on repetitive text, and 47.8 vs 46.1 on the ~15k-token prompt. The user notes the long-context row is a narrow win and that prompt processing under the tensor split is slower than their own layer split on x4 risers (850 vs 1,020 t/s).

generation:

50.0 tokens/s

quant:

Q6 (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Swift at a ceiling of 50 t/s on an RX 9070 XT 16GB and a Radeon AI PRO R9700 32GB. Setup is llama.cpp with the Vulkan backend, a Q6 quant, 131K context, and Q8 KV cache for both K and V, with a DFlash2 drafter. User asks whether others get more than 50 t/s on AMD 9070 XT or R9700 hardware with a usable context size, and whether a different OS could add 30+ t/s.

generation:

5-6 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at roughly 5–6 t/s on Strix Halo with max GPU offload. Setup is Windows + WSL2 + Docker, LM Studio with the Q4_K_M GGUF, Open WebUI frontend, and 262K context with context compaction enabled. The user calls the experience poor: refining a 12-slide deck repeatedly stalled for minutes and returned nothing across 3–4 retries, and asks whether the model, the 262K context, LM Studio, Open WebUI, or the stack is at fault.

generation:

20.6 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 NVFP4 at 20.6 t/s on an AMD Strix Halo 128GB, with 17.3 t/s on Windows. Setup is Atlas Inference running natively on HIP with custom DP4A math on gfx1151, using K=4 draft speculative decoding and a 65k agentic context window. The engine is the sole cross-architecture submission for MLPerf Inference v6.1 Edge Agentic LLM SingleStream, scoring 87.76 normalized BFCL and 84 on HA-20. Median TPOT is 47.3 ms and median TTFT is 2.7s across 65k context windows.

generation:

24.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks 12 quantizations of Qwen3.8-Flash-Next on an AMD Strix Halo 128GB, reporting 24 t/s decode and 312-382 t/s prefill at 512-2K tokens with llama-server. Setup is llama.cpp (llama-flash-next fork) with the Heretic2 IQ4_XS NGQ4 GGUF at 65,536 context, using Vulkan backend; the 92 GB model fits in 128 GB unified memory. Halogen Flash Server shows 2-4x faster prefill (453-1,203 t/s) but accuracy collapses to 65.2% without its proprietary overlay, and it forces thinking mode.

generation:

2.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at around 2 t/s on a local setup with 4 GB VRAM and 32 GB system RAM at an 8k context window. The model is offloaded to system RAM, which the user says makes generation barely usable. The user asks whether renting cloud GPUs with ~48 GB VRAM for $0.20–$0.30/hr and deploying Ollama or vLLM in Docker is a practical alternative for ad-hoc study.