Qwen3.8 reports

generation:

44.2 tokens/s

prompt processing (prefill):

1710.2 tokens/s

quant:

Q8_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on dual RTX 3090s, comparing llama.cpp against vLLM int8. llama.cpp with Q8_K_XL GGUF and q8_0 KV cache reaches 44.22 tok/s decode on narrative prompts, roughly half the 84.56 tok/s of the vLLM int8 w8a16 container with fp8 KV cache and MTP. Setup uses two RTX 3090s with a 0.55/0.45 split and ngl=99 for llama.cpp; vLLM uses custom all-reduce with P2P granted. Prefill at 10K tokens is 1710.16 tok/s for llama.cpp versus 1891.57 tok/s for vLLM. TTFT is about 5x slower on llama.cpp (337 ms vs 69 ms narrative). The 90K prefill run for llama.cpp was stopped early. User considers adding more system RAM for LMCache instead.

generation:

30.1 tokens/s

quant:

UD-IQ2_XXS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 30.10 tok/s median decode on an RTX 5060 8GB, over five runs at an occupied 8192-token prompt. Setup is llama.cpp with UD-IQ2_XXS weights and q4_0 KV cache, 65/65 layers on CUDA, peak VRAM 7767 MiB. A short-context FULL_GPU decode of 31.39 tok/s and a ctx512 control of 18.11 tok/s are also given; quality gate and uncensored checkpoint are not done.

generation:

149.0 tokens/s

quant:

INT8

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at up to 149 tok/s code decode on one RTX 4090. Setup is NInfer with INT8 prefill and E8 4-bit KV cache, full native 262K context, MTP3 speculative decoding, sm_89-retuned attention prefill, vision, and llama.cpp-compatible /metrics and /slots. User says it is their go-to even with Strata, which is limited to IQ2_XS.

generation:

10-20 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen3.8 Flash-Next Coder in a self-built Strata fork on a machine with 16GB RAM and 16GB VRAM, using NVMe, at around 10 to 20 tokens per second. No GPU model, quant, context length or KV cache is stated. The throughput is given only as a range, so no single figure is recorded. The user asks whether the fork should be published on GitHub.

generation:

125.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B with vision and MTP at 125 tok/s on an RTX 4090. Setup uses the integrated GPU for display output, freeing about 2.5GB of VRAM, which raised the context window from 65k to 132k. The user notes swapping the display cable back for gaming is a minor annoyance.

generation:

60.0 tokens/s

quant:

exl3-5.00bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~60 t/s on a single RTX 3090. Setup is ExLlamaV3 with an exl3-5.00bpw quant at ~150k context; prefill was not benchmarked. The machine has an Intel 12400 CPU and 32GB DDR4 RAM. User asks whether to buy 128GB DDR5 and a Ryzen 9700X to run the IQ3 quant under Strata, and how Qwen3.8 27B compares to Qwen Next on output accuracy for coding and Hermes assistant use.

generation:

59-68 tokens/s

quant:

GPTQ-Int4 (GPTQ)

kv:

turboquant_k3v4_nc

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~59-68 tok/s single-stream decode on 2x modded RTX 2080 Ti 22GB (SM75, TP=2, NVLink). Setup is a vLLM fork with GPTQ-Int4 self-quant, turboquant_k3v4_nc KV cache, 524,288 max-model-len (YaRN 2x), MTP K=2 speculative decoding, and vision enabled. Aggregate throughput is 364.7 tok/s at 16 lanes; 24 lanes regresses. The S4 scoped re-emission drafter measured 811.5 tok/s on copy-shaped spans but crashes at current HEAD, and v3 GPU-merge async is negative.

generation:

36.6 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 36.6 tok/s decode on a single modded RTX 2080 Ti 22GB, with 262,144 tokens of context. Setup is KVMem (retrieval-based long context) with IQ3_S weights, q8_0 KV cache, MTP speculative decoding (58.6%/61.6% acceptance) and vision, using 16,552 of 22,528 MiB VRAM. The user compares against the upstream author's RTX 5060 Ti 16GB at 31.7 tok/s, and notes the lossless KV-streaming alternative runs 30-42 tok/s in the resident window but drops to ~9.6 tok/s past 135K tokens; a 260,096-token needle-in-a-haystack test hit exactly.

generation:

39.9 tokens/s

quant:

8-bit (MLX)

kv:

4-bit / 8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 39.9 tok/s with DFlash 2 speculative decoding on an M5 Max 128GB, versus 17.9 tok/s plain decode. Setup is mlx-dspark 0.14 on MLX 0.32 with the 8-bit MLX build (~29 GB) and 4-bit/8-bit quantized KV cache; the DFlash 2 drafter is incoai/Qwen3.8-27B-DFlash2 (3.8 GB). In the agent harness the model runs 20-29 tok/s. A bf16 run of the same model measured 9.7 tok/s plain and 36.5 tok/s with DFlash 2 (4.43 accepted tokens per round, 137 target forwards, output identical). Agent-12 scores with DFlash 2 were 12/12 easy in 122 s and 7/8 hard in 401 s, about 3x faster than the Aug 18 run at the same accuracy. The Claude Code smoke test passed in 38 s. Gemma 4 31B Abliterated 4-bit ran at 26 tok/s and Gemma 4 E4B 4-bit failed the smoke test. On an M4 Pro 64 GB, Gemma 4 31B runs at ~13.5 tok/s and server-side prompt fixes cut hello latency from ~120 s to ~3-5 s.

generation:

18.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-NVFP4 at 18+ tok/s decode on an Intel Panther Lake Integrated Arc B390 with 32GiB UMA memory. Setup uses a custom inference engine with custom NVFP4 kernels targeting Intel Xe3, running on a custom operating system built by the user. The user notes Ubuntu with llama.cpp achieves less than half the decode performance. They also report ~6.5 tok/s decode with Nemotron3.5-Lightning-30B-A3B-NVFP4 on a Lenovo Yoga 7i (16GB UMA) requiring SSD streaming, with decode physically capped at 17 tok/s due to ~6G/s sequential reads.

prompt processing (prefill):

800.0 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User asks whether Swift 1.5 Flash-Next at IQ3_XXS will fit in 16GB VRAM and 32GB RAM, having run the IQ2_XS build on Strata at an average 50 t/s and 800 t/s prompt processing, sometimes up to 73 t/s. Setup is Strata with an IQ2_XS GGUF of the Flash-Next model. The user wants a higher quantization for lower KLD and better agentic coding accuracy and asks for others' configs.

generation:

10.9 tokens/s

prompt processing (prefill):

19.7 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares mainline llama.cpp against ik_llama.cpp on a hybrid 4-GPU rig with a Qwen3.8 Flash-Next 177B Uncensored IQ3_XXS model at 32k context, 35 layers offloaded to GPU and 13 on the CPU. Mainline reaches 19.70 t/s prompt eval and 10.86 t/s generation with 2,490 CUDA graphs reused; ik_llama.cpp drops to 5.48 t/s prompt eval and 8.43 t/s generation with no CUDA graphs and context checkpoints during generation. User asks whether ik_llama.cpp's advantage applies only to CPU or single-GPU setups, and whether its threadpool struggles with pipelined layer splits across heterogeneous GPUs over PCIe.

generation:

58.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 58 tok/s generation and 2,000 tok/s prefill on a Mac Studio M5 Ultra 256GB. Setup is MTPLX 2.12.0 with speculative decoding, 55 to 65 percent draft acceptance, 99 GB resident, 4k context. Four models run in slots on the same box; Gemma 4 26B 4-bit reaches 113 tok/s, Qwen 3.6 35B-A3B 114 tok/s, DeepSeek V4 Flash 4-bit 31 tok/s. DeepSeek prefill is from a single uncached 13,139-token request in 21.8 s. Clef-Flash 9B decision slot has a 259 ms median.

generation:

110.0 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 110+ t/s on a single RTX 5090 with a fork of llama.cpp called focus-llama. Setup uses Q6_K_XL GGUF quantization with q8_0 KV cache, 256k logical context capped to an ~85k physical KV buffer, and adaptive MTP speculative decoding. Speed ranges from about 120 t/s initially down to 60 t/s as context fills; the user cites 100-120 t/s depending on context fullness and says context usage stays at 18-30%.

generation:

18.0 tokens/s

prompt processing (prefill):

500-600 tokens/s

quant:

IQ3_S (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~18 tok/s decode and ~500-600 tok/s prefill at 64k context on an RTX 4070 Super 12GB with 32GB system RAM. Setup is stock llama.cpp with an ISTA-DASLab GSQ-RCO IQ3_S quant (~11GB), q4_0 KV cache, -ngl 58 and token_embd offloaded to CPU. Only 16 of 64 layers need KV cache, so 64k context takes about 1.1GB instead of 4GB at f16. The same recipe works with 0bserverx' Qwen3.8-27B-Heretic-GSQ-RCO IQ3_S quant at about a 7% speed loss. User calls GSQ-RCO the best Qwen3.8-27B quant tested and says it performs close to full precision.

generation:

32.6 tokens/s

prompt processing (prefill):

493.0 tokens/s

quant:

Q6_K (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Q6_K at 32.6 t/s decode (tg256, no MTP) on 2x Tesla P100 16GB with tensor split, versus 17.51 t/s upstream at the fork point. Setup is a llama.cpp fork with CUDA work for Pascal (sm_60), q4_0 KV cache, and fp16 math with fp32 accumulation. Prefill pp2048 at 0 context is 493 t/s versus ~250 t/s upstream. With MTP speculative decoding the fork reaches 54 t/s at 2k context and 29-35 t/s at 260k context. Prefill at 260k context is 123 t/s filling and 153 t/s for a question on a loaded context. Perplexity on the gate corpus at -c 4096 is 2.6101.

generation:

76.7 tokens/s

prompt processing (prefill):

792.4 tokens/s

quant:

MXFP4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B AXQ MXFP4 MTP at 76.70 tok/s decode and 792.4 tok/s prefill on an M5 Max 128 GB. Setup is AX Engine 7.5.3 with MXFP4 MTP weights, median of 20 runs on the flappy suite. The same campaign measured the 6-bit pack at 76.04 / 768.7 and long_code at 72.71 / 856.9; MTPLX 2.11.2 reached 63.91 / 697.3 and mlx-lm 34.20 on the MXFP4 pack.

generation:

63.5 tokens/s

prompt processing (prefill):

1119.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 63.5 t/s decode (code) and 1119 t/s prefill at 2.6K context on a Radeon Pro W7900 eGPU plus Strix Halo iGPU hybrid setup. Setup is llama-halo-hybrid fork with UD-Q4_K_XL quant at 128K context, MTP head enabled, dense layers on the W7900 and routed experts on the iGPU. Hybrid beats gufo iGPU-only (44.7 t/s decode code, 903 t/s prefill at 2.6K) and leaves 43-52 GiB RAM free versus 14-30. GLM-5.3-Flash UD-IQ3_XXS does 25 t/s (31.6 with MTP) and DS-V4-Flash UD-IQ3_XXS does 22.5 t/s. W7900 runs 100-104 C junction on long prefills with a locked 241 W power cap. One run each, no quality testing.

generation:

54.9 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 54.9 tok/s decode on an RTX 5070 Ti 16 GB, with 33 GiB of experts kept in RAM. Setup is EuLLM (llama.cpp b11370 + PR #29887) with IQ2_XS, 2048-token micro-batch, and a GPU expert cache; the model is 125B total with about 6B active per token. The same PC reaches 85.8 tok/s with Strata and MTP, and prompt processing peaks at 1,500–1,743 tok/s with prefetch versus 2,320 tok/s for Strata. MTP loses 27% on this model with 52% draft acceptance, and gains only +7% prose / +16% code on Qwen3.6-35B-A3B.

generation:

96.5 tokens/s

kv:

q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at a median 96.45 t/s decode on a 16GB RTX 5080, with a range of roughly 90-110 t/s during coding tasks. Setup is NInfer v1.5 with q4 KV cache, MTP speculative decoding with 3 draft tokens, 110,592 context allocated, and 11.86 GiB of weights leaving about 498 MiB of slack. Across 32 completed requests the lowest decode was 84.6 t/s and the highest 131.7 t/s, with median time to first token 1.4 seconds. About 68% of generated tokens were reasoning tokens, and one request that lost its prompt cache took almost 50 seconds to process a 79k prompt before generating at about 94 t/s.