Qwen3.8 27B at 4-bit generates a median of 42.5 tokens per second on the AMD RX 7900 XTX, the hardware it is most run on, from 6 plain runs (community reports, as of 26 Oct 2026).
As of 26 Oct 2026, Qwen3.8 27B at 4-bit on the hardware it is most run on, with the median of plain runs (one device, one request, no speculative decoding, the whole model in its memory):
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports about 40 tokens per second running a Swift-1.5 Qwen3.8 27B IQ3_S GGUF on an RTX 5060 Ti 16GB.
Setup is llama.cpp with MTP speculative decoding at 81920 context size, on a Ryzen 7 9800X3D with 32GB DDR5-6000.
User asks whether this is the best they can do for general use, coding and agentic tasks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 58.53 tok/s on an RTX PRO 4000 Blackwell SFF, up from 35.04 tok/s with DFlash2 speculation after power and clock tuning.
Setup is NVFP4 weights at 8,000 input tokens with ECC off, best at a 120 W limit, VRAM +20% and core +180 MHz; the baseline was 70 W with zero offsets.
A second model, Qwen3.6 35B A3B in Q4_K_S with no drafter, went from 84.97 to 115.56 tok/s at 120 W, VRAM +33.3% and core offset 0. The user notes 120 W exceeds the 75 W PCIe slot budget on this connectorless card and calls it an experiment, not a recommended daily profile.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-uselong-context
User reports Qwen3.8 Flash-Next (Swift 1.5) at 144.6 tok/s decode and 3,444 tok/s prefill on a Mac Studio M5 Ultra 96GB, at a median context of 147.5K tokens.
Setup is Splash with IQ3_S GGUF weights and a 262,144-token context window, driven by the DeepSeek Harness agent framework for agentic coding.
With two concurrent requests combined decode is 200.4 tok/s (100.2 per request) and prefill 3,006 tok/s; with three it is 218.0 tok/s combined (72.7 per request) and 3,253 tok/s prefill, a result the user calls preliminary. The user is happy with the numbers and finds the model less lazy than Sol at troubleshooting.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B generation rising from 35.04 to 58.53 tok/s (+67%) on one RTX PRO 4000 Blackwell SFF after raising the power limit from 70 W to 120 W with VRAM +20% and core +180 MHz offsets.
Setup used a modified VBIOS loaded into memory via a patched loader and NVIDIA open kernel modules, with no EEPROM flash; headline comparisons used 8,000 input tokens with ECC off.
Only 1-3 requests per configuration across 158 total, and the 70 W baseline used the same modified VBIOS rather than factory firmware. Pushing VRAM from +20% to +33.3% slightly reduced 27B generation, 57.70 to 56.91 tok/s. Qwen3.6 35B A3B was almost flat between 106 and 120 W.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 2076 tok/s prompt processing on a 145k-token prompt and 95-105 tok/s decode depending on task, with peaks of 145 tok/s, on a single RTX 4090 24 GB.
Setup is Strata 0.1.43 with an IQ3_S pack, int8 KV cache, 204800 context, resident experts and MTP speculation, on a Ryzen 9 3900X with 48 GiB DDR4-3200 under Windows 10.
A grid search over flags raised decode from 97 to 142 tok/s; the best arm was --kv-resident 20480 plus --adapt-async 1 and STRATA_ADAPT_LAG=2, giving 1526 and 2309 tok/s prompt processing at 10k and 20k. The user notes Windows does not pin memory with CUDA, so every host-to-GPU expert copy is CPU-driven.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodinglong-context
User benchmarks Qwen3.8 27B (Swift-1.5, AWQ-INT4) on dual RTX 3080 20GB cards in tensor parallel, reporting 2080 t/s prefill at 8192 prompt tokens with gpu-unlock plus P2P versus 967.1 t/s stock, a 115% gain.
Setup is vLLM 0.32.0 with FP8 KV cache, MTP speculative decoding with 3 tokens, and both cards limited to 240W in PCIe 3.0 x16 slots.
Prefill gains range from 70% to 115% across prompts of 8192 to 261900 tokens. Decode is reported as inter-token latency step rates of about 15 to 16.6 steps/s, which the user says corresponds to roughly 45 tok/s at an MTP acceptance length of 2; decode changes are within noise except at 261k tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8-27B at 66 t/s decode at 262,144-token context depth on one RTX 3090, with the full context fitting in 23.5 GB.
Setup is a custom engine with unsloth UD-Q4_K_XL weights, K cache at 6 bits, V at 5 bits, and the newest 2,048 positions kept at q8_0, saving 1.1 GB versus a full q8_0 cache at 120k context.
KV quality was scored by KL divergence against llama.cpp q8_0 output: 0.00160 at 64k versus 0.00163 for q8_0 and 0.00161 for fp16, +8% at 32k, and the same 230/282 on the author's question set. The author notes Qwen3.8 is a hybrid with only 16 of 64 layers attention, so full-attention models may differ.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B ASCII-Condensed UD-IQ4_XS at about 81 t/s code generation on an RX 9070 16 GB card with 32 GB system RAM, running on Windows.
Setup is achyudh's llama.cpp ROCm KV-streaming fork with q8_0 K / q4_0 V cache, 262144 context, --kv-stream with 300/1800 MiB resident KV, MTP n=4 plus n-gram speculation, and 6 threads.
Decode was 75 t/s at 9k and 95 t/s at 21k tokens, prefill about 1000+ t/s; these are single runs on short prompts, not filled 64k-262k contexts, and output quality was not checked. The user notes their numbers are below the fork README's ~100+ t/s at 16k-64k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports about 30 tok/s with the Swift-1.5 version of Qwen3.8 Flash-Next on a Tesla P100 16GB in a PCIe3 x16 slot, after switching from llama.cpp to Strata.
Setup is Strata with 64GB of ECC DDR4 at 2100 and a 131072-token context; the logs show expert cache hits of 63.5% to 77.8% with routed experts going over PCIe, so part of the model ran from system RAM.
Individual runs in the pasted logs range from 28.7 to 33.3 tok/s on prompts of roughly 78k tokens, which were read in 2 to 3 s. The user says it serves their Hermes agent and calls it impeccable for a local LLM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingcreative-writing
User benchmarks Qwen3.8-27B on two free Kaggle T4s (no P2P, 70W cap per card) with EXL3 4.0bpw under exllamav3, tensor parallel over PCIe, and a DFlash2 block drafter of 7 tokens.
Overall throughput is 55-57 tok/s with drafting versus ~18-19 without; HumanEval reaches 154 tok/s and prose 35-37 tok/s.
Prose acceptance is the limit at about 2.5 accepted tokens per step versus ~7 on code. On the same cards llama.cpp best was 16.5 tok/s and vLLM int4 with its own MTP drafter did 40.8 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks a llama.cpp PR that fuses FFN gate/up matmuls and GLU in MMQ, on a DGX Spark (GB10) and an RTX 5090.
Setup is llama.cpp with llama-bench, -ngl 999, flash attention on, 16384-token prompts, F16 KV cache, batch 2048, 5 repetitions.
On the DGX Spark, Qwen3.8-27B Q4_K_M prompt processing rises from 840.1 to 899.9 tok/s (+7.1%). On the RTX 5090 the same model and quant goes from 3688.4 to 3790.9 tok/s. The table also covers Qwen3.6-27B and Llama-3.1-8B across Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS; only prompt processing was measured (-n 0).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagenticlong-context
User reports Qwen3.8-Flash-Next at 49 t/s decode on an M5 24GB MacBook Air.
Setup is Splash 1.4.0 with a small fork, ISTA GSQ-RCO Q2_0 weights and a Flash-Next-DFlash2 draft model at 256K context. About 28% of expert slots (6,800 of 24,576, ~8.8 GB) stay resident and the rest are streamed from the SSD.
Cold prefill without prompt cache is 522 t/s on a 32,620-token prompt and 485 t/s on 63,201 tokens. Real agent use in pi averages ~45 t/s with no prompt-cache losses, and draft acceptance is ~.70 for code and ~.80 for tool calls.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 164.6 tok/s decode on an RTX 5090 32GB, with prefill of 6,000~6,500 tok/s.
Setup is Strata with an IQ3_S GGUF quant, on a Ryzen 9 9950X3D system with 64GB DDR5.
The user previously ran IQ3_XSS and switched to IQ3_S for more speed and quality. The prompt asked for a Minecraft-style Eiffel Tower, which the model built in detail.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target plus DFlash 2 draft) on a MacBook Pro M5 Max with 48 GB, using a Python code context and 128 generated tokens per test.
At a 4096-token prompt the single request reaches 93.8 t/s decode and 1207.0 t/s prefill; longer prompts give 189.4 t/s decode at 16382 tokens and 110.2 t/s at 65536 tokens.
Batched continuous batching reaches 413.7 t/s aggregate at 2x and 407.0 t/s at 4x, with peak memory between 19.1 GB and 22.2 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 125B at 20.0 tok/s decode on an RTX 4090 at the full 262k context window, with experts streamed from NVMe SSD.
Setup is ExLlamaV3 with a 3.05bpw_h5_ng5 quant and Q4 KV cache, 16 GB VRAM and only 8 GB system RAM allocated.
At 100k context the same card gives 20.8 tok/s decode and ~1,536 tok/s prefill; short 8k context gives 28-33 tok/s prose and 19-22 tok/s code with ~1,080 tok/s prefill. An RTX 3080 Laptop with 16 GB VRAM and 32 GB RAM reaches ~18-21 tok/s decode and ~650-700 tok/s prefill at 100k context, peaking at 15.5 GB VRAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-Flash-Next 180B-A6B at 46-53 tok/s generation and ~770 tok/s prompt processing on a Strix Halo mini PC with 128 GB unified memory.
Setup is llama.cpp and later the Strata engine, with UD-IQ4_XS GGUF weights in 3 shards at 94 GB, plus MTP speculative decoding and a vision projector.
The same box ran Qwen3.6-35B-A3B at 76.5 tok/s, and that model did 11.7 tok/s CPU-only on the user's Ryzen 7 6800U laptop. The 46-53 tok/s figure is a range, so no single generation rate is recorded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Qwen3.8-Flash-Next IQ3_S on an RTX 5080 16 GB with a Core Ultra 9 285K and 62 GB DDR5-6800, averaging 154.9 t/s generation and 5,475 t/s prompt processing in a context sweep of fresh 15K-207K prompts.
Setup is Basalt with IQ3_S weights, 262K context and int8 KV cache, using 15.7 GB VRAM and 54.1 GB system RAM.
An agent coding session over 103 requests reached 153.0 t/s average generation with a 262 t/s peak, and 5,011 t/s average prefill; 99.5% of prompt tokens came from cache with a 0.2 s median time to first token.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at an average 208.2 tok/s on a single RTX 5090 32GB.
Setup is SGLang in Docker with the gittensor NVFP4 build and the DSpark speculative drafter, served over an OpenAI-compatible API.
Five runs of 256 tokens with thinking on ranged from 148.3 tok/s (cold start) to 232.4 tok/s, averaging 208.2; the user says it sits in the 220s once warm and peaks above 230.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B averaging 95 t/s decode and 1500 t/s prefill on a single RX 7900 XTX using a hipfire fork.
No quant, context length or KV cache settings are stated.
The user calls the result a removal of AMD's governor and is enthusiastic about it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 353 t/s on four V100 16GB cards.
Setup is NInfer with NVFP4 weights, Q8 KV cache, 256k max context and MTP 4/0 speculation, tensor parallel across 4 GPUs.
The user previously ran 2 V100s at 181 t/s and notes the repo now supports tp4.
Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.
The hardware Qwen3.8 27B at 4-bit is most run on: AMD RX 7900 XTX (42.5 t/s median, 6 runs); NVIDIA RTX 3090 (38.4 t/s median, 5 runs); NVIDIA RTX 5090 (73.2 t/s median, 5 runs). Medians of plain runs, as of 26 Oct 2026.
Which inference engines do people use for Qwen3.8?
Reports of Qwen3.8 name llama.cpp (406 reports), Strata (127 reports), vLLM (111 reports), NInfer (44 reports), as of 26 Oct 2026.
Speed by hardware on Qwen3.8 27B at 4-bit
The size and quant level the most devices have plain runs of: one device, one request, no speculative decoding, and the whole model in the device's memory. Every speed in this table is the same model.
With part of the model in system RAM: Qwen3.8 125B · 6B active at 4-bit
Runs on one device that kept some of the model in system RAM or streamed it from an SSD. The CPU and its memory set the pace as much as the card, so these figures are kept apart from runs with the whole model on the device and vary more from machine to machine.