Qwen3.8 reports

generation:

36.9 tokens/s

quant:

Q8 (Splash)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 36.9 t/s average on an M5 Pro 64GB using the Splash engine with native 8-bit weights. Setup is Splash with speculative decoding, 27 GB native 8-bit model, evaluated at temperature 0.0 across five task domains; speeds range from 21.9 to 54.8 t/s. Context scaling tested to 190,016 tokens with decode speed staying between 21 and 33 t/s; cold prefill at 180k+ tokens takes 4-5 minutes.

generation:

37.0 tokens/s

prompt processing (prefill):

1069.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 37 t/s generation and 1069 t/s prompt processing on two Radeon AI PRO R9700 32GB cards in a GMKtec EVO-X2 Strix Halo machine. Setup is llama.cpp with the ROCm build, RCCL enabled via -DGGML_HIP_RCCL=ON, and the UD-Q4_K_XL GGUF. The GPUs connect over NVMe-to-USB4 enclosures and run at PCIe 4.0 x4. Tensor parallelism was slower than a single card under the Vulkan build; the ROCm build restored a decode speedup, and RCCL restored full prompt processing. Single card measured 28 t/s generation and 1070 t/s prompt processing.

generation:

50.0 tokens/s

quant:

exl3-5.0bpw (EXL3)

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 50 t/s on an RTX 5060 Ti 16GB plus RTX 3060 12GB, with MTP enabled the rate varies between 40 and 60 t/s. Setup is ExLlamaV3 through TabbyAPI with the turboderp exl3-5.0bpw weights, 102,400 tokens of context, a Q8 KV cache and Q8 draft cache, tensor parallelism across the two cards, and 32GB of system RAM. Without MTP the rate drops to 22 t/s. The user notes 1.3GB free on the 3060 and 0.6GB on the 5060 and is trying to rebalance the GPU split.

generation:

70.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

W4A16 (AutoRound)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at roughly 2,000 t/s prefill and 70 t/s decode on 4x Radeon Pro V620 32GB. Setup is Intel Autoround W4A16 weights running locally with the OMP harness, which the user believes made a big difference. The model ran for around 3 hours, mostly driving two browsers at once to test and fix a generated HTML/JS 3D game.

prompt processing (prefill):

250.0 tokens/s

quant:

Q4

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B running a local agent loop for about 21 days on a single RTX 3090, with prefill around 250 t/s. Setup is vLLM serving the Q4 model with a Q8 KV cache at 200k context, using a deepseek harness and a written rulebook; llama.cpp reaches about 700 prefill t/s on the same card. The run used 180 subagents, about 230M tokens in and out, 1.7B cache-read tokens, and 699 compactions totaling about 83 hours, roughly 17% of calendar time, with a typical compact taking about 7 minutes on a 160k+ token prompt.

generation:

80.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

3bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80 t/s generation and 1500 t/s prefill on 3x RTX 3090 with 128GB DDR4. Setup is ExLlamaV3 with an exl3 3bpw quant at 262,144 context, with vision and speculative decoding enabled. A single RTX 5090 with 128GB DDR4 reached 29 t/s generation at the same 1500 t/s prefill and 262k context. User plans to test 4bpw later and says ExLlamaV3 is replacing vLLM and llama.cpp for this model.

generation:

29.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B GSQ-RCO-IQ3-XXS at 29 t/s on an RTX 3060 12GB at full context, and 40 t/s at near-empty context. Setup is llama.cpp with MTP speculative decoding and high thinking enabled, using a 122K context window. The run used roughly 75K tokens and the context was compacted twice; the user rates this as the most impressive local model in a nine-model web-dev comparison.

generation:

45.0 tokens/s

prompt processing (prefill):

600.0 tokens/s

quant:

Q4

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 45 t/s generation and 600 t/s prompt processing on a dual RTX 3060 12GB setup. Setup uses a Q4 model and Q4 KV cache at roughly 100k context, with the model running fully on the GPUs. The user is asking for upgrade advice and considering a 4x RTX 3060 build or 24GB Intel cards, and wonders whether PCIe lanes would bottleneck a four-card setup.

generation:

40.0 tokens/s

quant:

Q6_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 40+ t/s on two Tesla P100s, with a roughly 30 t/s average across long contexts and up to 55 t/s at 0 context. Setup is a custom llama.cpp fork with P100 kernel optimizations, Q6_K quant, both cards capped at 175W and communicating over PCIe gen 3. Speeds vary by about ±2 t/s; the user notes fp16 math saves 40% or more on prefill with negligible accuracy loss.

generation:

147.0 tokens/s

quant:

W4A16 (W4A16)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 147.0 t/s single-stream decode on one NVIDIA CMP 170HX 64GB, with 134.7 t/s at 4K, 100.1 t/s at 65K, ~90 t/s at 126K, and 64.9 t/s at 250K context. Setup is vLLM 0.27.1 with a W4A16 target, a DFlash2 W4A16 drafter, FP8 target KV, BF16 draft KV, a custom SM80 split-KV verifier, full CUDA Graph, 35 verifier segments / 140 CTAs, k=3 draft tokens, MAX_SEQS=1, and 1350 MHz / 180W. These are decode-only numbers, not end-to-end throughput including prefill. FP8 beat INT8 at long context (48.3 vs 115.4 ms/iter at 250K). Allowing 2 concurrent long requests made makespan and slowest-request throughput worse at both 126K and 250K.

generation:

60.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~60 t/s on a rented RTX Pro 6000 Blackwell 96GB. Setup uses the Q4_K_M GGUF, which the user notes is not efficient for this card but was chosen because abliterated models that fit the memory are limited. The rental costs about $30/day on Vast, and the user is asking which hardware configuration would offer the best price and versatility for future model generations.

generation:

202.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B token generation rising from 110 T/S to 202 T/S on a CMP 170HX 40GB after overclocking. Nothing changed except the overclock, which raised memory bandwidth from 1,386.2 GB/s to 1,890.1 GB/s, a +36.4% increase. GPU wattage is 300 watts and GPU temps are slightly lower after the overclock.

generation:

110.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

3.05bpw (EXL3)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 110+ t/s generation and over 1,500 t/s prompt processing on 3x RTX 3090s. Setup is ExLlamaV3 with a 3.05 bpw EXL3 quant at full context and FP16 KV cache, with only the engram table offloaded to RAM. All three cards run at 250-265W and one is on a TB4 eGPU. User also ran Qwen3.8 27B (UD-Q8_K_L) on 2x 3090s in TP and a UD-Q4_K_XL of Qwen3.8 Flash Next that partially offloaded to RAM, reaching about 50 t/s max output, 38 t/s average, and prompt processing in the low hundreds.

generation:

43.4 tokens/s

prompt processing (prefill):

4679.0 tokens/s

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Next running on a 6x V100 16GB setup with TP2 PP3, reaching up to 4,679 t/s prefill at 16K input and 43.38 t/s generation with MTP enabled. Setup uses the 1cat-vllm engine with speculative decoding limited to speculative=1; KV cache is 8.78 GiB, 531,288 tokens, supporting 2.03x concurrency at 262,144 tokens per request. MTP nearly doubles generation throughput versus base (22.59 t/s average without MTP). A 20-minute gpu-burn stress test peaked at 65°C and stabilized at 64°C with fans at 76%.

generation:

50.0 tokens/s

prompt processing (prefill):

2300.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 50 t/s generation and 2300 t/s prompt processing on a single RTX 5090 with 128 GB DDR5. Setup is FreeToken with the nvidia/Qwen3.8-Flash-Next-NVFP4 model converted to FreeToken format, expert caching enabled, max sequence length 229376, and max running requests 2. Generation speed ranges from 40 to 60 t/s depending on cache effectiveness. User notes llama.cpp lacks MoE expert caching, and FreeToken is early with no KV quantization or MTP yet.

generation:

39.9 tokens/s

prompt processing (prefill):

1376.9 tokens/s

quant:

Q6_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 39.88 t/s decode and 1,376.87 t/s prompt processing on a mismatched pair of Tesla V100 PCIe cards (16 GB + 32 GB, 48 GB total). Setup is mainline llama.cpp with the Q6_K_M GGUF, tensor split, Flash Attention, Q8 KV cache, batch/ubatch 2,048, 20 threads, and NUMA distribute, with the 32 GB V100 as main GPU. Prompt processing measured 1,324.34 t/s at 4k and 1,221.49 t/s at 16k. A Qwen3.8 Flash Next 177B/6B-active MoE Q4_K_XL GGUF ran at 26.29 t/s decode and 380.67 t/s prompt on the same two cards.

generation:

38.3 tokens/s

prompt processing (prefill):

937.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 38.3 t/s decode and 937 t/s prefill on a Ryzen AI Max+ 395 with 128 GB, at 1,004,581 tokens of context. Setup is the halogen server (0.12.0) with HALOGEN_ROPE_YARN=4 and HALOGEN_CTX=1048576, greedy, 64 tokens, using the default speculative drafter; the 1M row is a single cold request. At 258,794 tokens the same setup gives 45.0 t/s decode and 1,114 t/s prefill; a follow-up turn over the prompt cache at 1M reaches first token in about 0.55 s.

generation:

15.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 15 t/s on an RX 7900 XT with 20 GB VRAM and 96 GB DDR5 RAM. User asks whether to add another 96 GB of system RAM or a second RX 7900 XT for running local LLMs such as Qwen3.8 27B or Qwen3.8 Flash Next.

generation:

83.6 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B IQ3_XXS against Bonsai Ternary PQ2 on a 16GB VRAM GPU. Setup is llama.cpp with MTP speculative decoding for Qwen; Bonsai lacks MTP. Qwen completed the agent suite in 8:00 versus 24:09 for Bonsai, using 27,197 output tokens versus 84,176. Weighted decode was 83.59 tok/s for Qwen and 64.91 tok/s for Bonsai. User notes the test is not scientific and prefers the IQ3_XXS visual results.

generation:

30-35 tokens/s

prompt processing (prefill):

350.0 tokens/s

quant:

Q6_K (GGUF)

kv:

K: Q8_0, V: Q5_1

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at around 30-35 t/s on two CMP 50HX mining cards (20GB + 10GB, 30GB total) with MTP enabled. Setup is llama.cpp with a Q6_K GGUF at 130k context and K Q8_0 / V Q5_1 KV cache, GPUs capped at 180W. Prompt processing is around 300-400 t/s. A 256k context preset with Q8_0 KV cache runs at 10-14 t/s with prompt processing around 40-50 t/s. Ornith 1.5 35B-A3B (Heretic-MTP-APEX-I-Balanced) reaches 80-100 t/s. A Q4_K_M Unsloth quant with 1,1 tensor split and MTP gives around 40 t/s.