Qwen3.8 reports

generation:

99.0 tokens/s

prompt processing (prefill):

1812.0 tokens/s

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 99 t/s single request and 1150 t/s batched on an RTX 3090. Setup uses a hyper-optimized inference engine with an fp8 KV cache, int8 lm_head and embeddings, fp16 recurrent state, and int8 activations, at 64k context in fast mode. Prefill reaches 1812 t/s at 1k context and 1000 t/s at 100k context.

generation:

22.4 tokens/s

prompt processing (prefill):

360.2 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B at 22.38 t/s generation (tg128) and 360.22 t/s prompt processing (pp512) on a Mac Studio M2 Ultra with 192GB unified memory. Setup is llama.cpp build b10450 llama-bench with the Unsloth UD-Q6_K_XL quant (24.13 GiB), flash attention on, q8_0 K and V cache and batch and micro-batch 2048. The user serves the model through llama-server at 131072 context with MTP speculative decoding (draft-mtp, draft max 3), and gets around 16 t/s on the WebUI with a flappy bird game prompt.

generation:

96.8 tokens/s

quant:

groupwise-int

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a long coding-agent session with DeepSeek Harness at a weighted 96.8 t/s streaming rate and a median of 104.4 t/s. The model is a groupwise-int artifact, not NVFP4. Context compaction worked well.

generation:

20.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports building a game in two prompts with Qwen 3.8 on 2x RTX 3060. Setup is llama.cpp with the UD-Q4_K_XL quant, 128k context, and a quantized cache at k5_0/v4_1. The user notes that 32 GB of system RAM caused issues after context compaction, which a 'continue' resolved.

generation:

50.4 tokens/s

prompt processing (prefill):

226.8 tokens/s

quant:

NVFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a custom iMatrix quant with mixed precision: NVFP4 for bulk matrices, Q5_K/Q6_K for attention, DeltaNet and FFN, Q6_K embeddings, Q8_0 output head and NVFP4 MTP. WikiText-2 PPL is 6.1197 versus 6.1127 for Q4_1. MTP target-only decode runs at 21.189 t/s, with MTP at 59.456 t/s. At full 261.5K context decode is 12.606 t/s. The GGUF is published on HF.

generation:

18-20 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~18-20 t/s generation speed on 16 GB VRAM and 64 GB system RAM. Setup uses FFN offloading to keep attention on the GPU, a q8_0 KV cache at 65k context, MTP speculative decoding, and thread pinning.

generation:

169.7 tokens/s

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports median decode speeds by context size on an unspecified setup: 169.7 t/s under 50k, 167.6 t/s at 50-100k, 156.9 t/s at 100-150k, 149.0 t/s at 150-200k, and 143.7 t/s at 200k and above. Context length is set to maximum. Peak decode for a single request is 222.0 t/s, with 211.2 t/s sustained over 5s.

quant:

IQ4_XS (GGUF)

kv:

Q4

User benchmarks Qwen3.8-27B on 16 GB of VRAM across several quants, including IQ4_XS and AD-IQ3_S-IQ3_XXS, and KV cache formats including kvarn4. The recommended config is Qwen3.8-27B-IQ4_XS-pure-MTP.gguf with beellama.cpp, a kvarn4 KV cache, MTP speculative decoding, and 32k-48k context. Quality metrics are 92.55% Top-1 match and 0.1169 KLD against Q8_0. The user also references Qwen3.6-27B quantizations from an external source.

generation:

56.0 tokens/s

prompt processing (prefill):

3000.0 tokens/s

quant:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 56 t/s decode and about 3K t/s prefill on a single RTX PRO 6000 96GB GPU. Setup is vLLM with an 8-bit quant at about 262K context, serving three concurrent users. The figures are for three concurrent users, and the user offers free community access rather than presenting a benchmark.

generation:

54.1 tokens/s

prompt processing (prefill):

1522.4 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a full 256k context offloaded to VRAM with no CPU spillover, using n-gram speculative decoding. Generation speed degrades with context depth: 43.69 t/s at 32k, 36.59 t/s at 64k, 27.55 t/s at 128k, 22.15 t/s at 192k, and 18.48 t/s at 256k. Prompt processing also degrades similarly.

generation:

50-60 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 50-60 t/s on two RTX 5060 Ti 16GB cards. Setup is LM Studio with Q4 quantization and MTP. A comparison to Qwen 3.6 gave 30 t/s with MTP, with an OS change from Windows to Linux. The user also downloaded NVFP4 but did not try it in vLLM.

generation:

104.8 tokens/s

prompt processing (prefill):

12400.0 tokens/s

quant:

groupwise-int

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B on an RTX PRO 6000 Blackwell at 104.83 tok/s weighted decode, running a long-horizon agentic task at 262K context. Setup is NInfer with a groupwise-int quant (mixed Q4/Q5/Q6). The run covered 966 model calls, 131.2M input tokens and 853.3K output tokens. Prompt processing was about 12.4K tok/s, measured client-side and including queue time. The user plans to try an NVFP4 profile and estimates API cost savings of $650+.

generation:

36.0 tokens/s

prompt processing (prefill):

560.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports llama.cpp RPC across a 5070 Ti and a 1080 Ti over gigabit ethernet, running Qwen 3.8 27B at UD-Q4_K_XL with a Q8 KV cache and 96k context. Best generation was 36 t/s with MTP enabled and GPU order RPC0,CUDA0. A prefill-focused config gave 560 pp and 19 tg. UD-Q5_K_XL with a full KV cache gave 380 pp and 15 tg. User notes MTP tanks prefill speeds and uses VRAM, and that enabling KV quant for the draft model reduces context. User recommends jumbo frames and careful GPU ordering.

generation:

80.3 tokens/s

prompt processing (prefill):

982.5 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 in llama.cpp with MTP combined with ngram-mod speculative decoding on a coding task that repeats a code block 10 times. Output came in bursts, starting at up to 169 t/s and degrading over the repetitions to about 80 t/s average. Prompt eval was 982.53 t/s and generation 80.30 t/s, with draft acceptance 0.85192. User asks whether parameter tuning would help and whether the ngram combination is worth it.

generation:

45.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Q4_K_M at 1M context on an AMD Strix Halo 128GB with an eGPU RTX 3080 Ti 12GB over Oculink, reaching about 45 t/s. Setup splits layers: 2GB of weights and FastMTP on the 3080, the rest on the iGPU. At 262K context, Q6 gives 10 t/s on the Strix alone, 24 t/s with MTP n=4, and about 28 t/s with FastMTP offloaded to the 3080. Q4_K_M at 262K context with 5GB of weights and FastMTP on the 3080 gives about 53 t/s. The user notes 60-70GB free on the AMD, enough to load an embedding and reranker, and compares to Qwen3.6-35B on 4x3090 at 450 t/s.

generation:

30.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running a local LLM on Alibaba's XuanTie C950 RISC-V CPU. No GPU is mentioned in the setup.

generation:

36.0 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B generating at 36 t/s while building a shooter game. The user mentions bugs in the result but is positive overall.

generation:

120.0 tokens/s

quant:

Q5_K_L (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DFlash2 speculative decoding with Qwen3.8 27B on an RTX 5090, with bursts of ~200 tk/s on code, ~80-90 tk/s on thinking, and an average of ~120 tk/s for code generation. Setup uses a Q5_K_L base model and a Q4_K_M draft model. The user compares this to MTP at ~140 tk/s on code and ~100 tk/s otherwise, notes the setup is memory hungry and required reducing context from 220k to 160k, and reports better results previously with DFlash on Qwen3.6 27B.

generation:

30.0 tokens/s

prompt processing (prefill):

1000-1300 tokens/s

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at Q8 running at 30 t/s with 1-1.3k t/s prefill on a Threadripper 3975WX with four Radeon Pro V620 cards. User measures electricity rather than speed: about 0.8-0.85 kW during inference and 0.15-0.2 kW idle, which at 0.32 dollars per kWh works out to roughly 55-60 dollars a month for six hours of inference a day. User calls 30 t/s pretty slow and reports 3.5k prefill and 70-80 t/s on Qwen3.6 35B instead.

generation:

26.9 tokens/s

prompt processing (prefill):

299.3 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a llama.cpp RPC cluster with tensor-split 20,23. Setup uses MTP speculative decoding with 84.37% draft acceptance and a 96.14% prefix cache hit ratio. Max context was stress-tested at 72,712/77,000 tokens.