Qwen3.8 reports

generation:

50.0 tokens/s

prompt processing (prefill):

1000.0 tokens/s

quant:

IQ1_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next Coder at 50 t/s generation and 1k t/s prefill on 2x RTX 3060 12GB. Setup is Strata engine with IQ1_M quant, 48 GB system RAM, Ryzen 7 5700X. User built a voxel pagoda as a first test and asks what others are building.

generation:

151.3 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

D' (K 6-bit, V 5-bit, 8-bit ring)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 151.3 tok/s greedy decode on short prompts on one RTX 3090, using a fork of the open-jet engine with the model's own MTP head (7 drafts) and a custom D' KV cache (K 6-bit, V 5-bit, 8-bit ring for the newest ~2.5k tokens). Setup is the open-jet fork with UD-Q4_K_XL weights at 120k context, single stream, one request at a time. The same engine with q8 KV gives 150.2 tok/s; llama.cpp tuned with MTP 4 drafts and q8_0 KV gives 82.7 tok/s on the same card. Sampled thinking runs at 134.9 tok/s, decode at 38k context at 114.3 tok/s, and at 101.6k context at 105.4 tok/s. The full 262,144-token context fits on the card at 23.4 GB, reading 260,896 tokens in 334 s (781 tok/s) then decoding at 63 tok/s. Greedy token ids are identical at draft depths 0, 4 and 7, so output is unchanged with speculation off. The user notes the card is an ASUS Strix 3090 with a raised power limit (~470 W) and memory at 9,851 MHz, and that numbers will differ on other hardware.

prompt processing (prefill):

8535.0 tokens/s

quant:

NVFP4 GPTQ (GPTQ)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 146.5-152.5 t/s generation and 8,535 t/s prompt processing on an RTX 5090 32GB with 128GB RAM. Setup is Strata with NVFP4 GPTQ 4-bit weights and 128K context; the model is a 125B-A6B MoE with speculative decoding and expert offload to RAM and SSD. The figures are a range over three runs; an Unsloth UD-Q4_K_XL run on the same box gave 132.7-138.8 t/s generation and 6,255 t/s prompt processing. The post is a beginner guide and also cites official and community numbers for other hardware.

prompt processing (prefill):

2013.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next prefill at 2,013 tok/s on a Ryzen AI MAX+ 395 (Strix Halo 128GB) using the Gufo engine with UD-IQ4_XS quant. Setup uses GPU + XDNA2 NPU prefill fork, with MTP and caches on for the served 42k cold run at 1,836 tok/s. Prefill scales from 1,837 tok/s GPU-only to 2,013 tok/s with NPU at 8,192 tokens, and from 1,788 to 1,949 tok/s at 32,768 tokens.

generation:

51.3 tokens/s

prompt processing (prefill):

2372.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 2,372 tok/s prefill on a 16K prompt and 51.3 tok/s decode at 8K context on Strix Halo 128GB. Setup is the hipfire 0.4.2.1 engine with MTP speculative decoding on, the XDNA2 NPU handling trunk input projections for two prefill banks alongside the iGPU. Prefill falls to 1,865 tok/s at 128K cold and 1,589 tok/s at 256K cold; decode is 41.8 tok/s at 32K, 42.1 at 128K and 35.6 at 256K. Numbers were measured on a 7.3-rc7 kernel with the IOMMU PerfOpt series, and the user expects lower prefill on stock kernels.

generation:

60.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 60 tok/s on a Strix Halo 128GB laptop (ASUS ProArt PX13, Ryzen AI Max+ 395, 96 GB set aside for GPU). Setup is Gufo (Thomas9120's Windows fork) with UD-Q4_K_XL quant and Q8_0 MTP draft model, 262144 context, speculative decoding with 7 draft tokens and prompt lookup enabled. Figures come from an 80-turn overnight coding run going from 45K to 262K context, not a controlled benchmark. User notes prompt lookup adds 20-30% on code-edit turns and the 10.16 draft changes add another 20-25%.

generation:

162.9 tokens/s

quant:

NVFP4 GPTQ (GPTQ)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-Flash-Next NVFP4 GPTQ against Unsloth UD-Q4_K_XL on an RTX 5090 32GB with 128GB DDR5-6000 RAM and an Intel Core Ultra 7 270K Plus, running native Strata inference on Windows. Setup is Strata 0.1.42-nvfp4.2 with 128K context capacity, INT8 KV cache, MTP 6, 15 CPU workers and a 1600MiB VRAM reserve. NVFP4 reached 162.9 t/s at 4K, 161.3 t/s at 32K and 154.1 t/s at 128K; Unsloth UD-Q4_K_XL reached 130.5, 131.0 and 127.6 t/s respectively. Time to first visible answer was 1.42/3.90/14.90 seconds for NVFP4 and 1.99/5.24/20.41 seconds for Unsloth. Each figure is the median of three runs on identical synthetic code-review prompts with 256 output tokens, temperature 0 and reasoning disabled. NVFP4 figures come from its best complete session of four, which the user notes favours NVFP4. In quality testing Unsloth passed 73/82 selected tasks versus NVFP4's 70/82, mainly through instruction following; maths and coding tied.

generation:

14.6 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 14.6 tok/s on a Colab L4 GPU. Setup is llama.cpp with Q4_K_M weights, one pass at the preset's defaults. The same model measured 47.2 tok/s on an A100 40GB and 71.2 tok/s on a G4 (RTX PRO 6000). The post also lists EXL3 4.0 bpw runs (16.3 / 47.5 / 65.3 tok/s) and Qwen3.8-Flash-Next IQ2_XS under Strata (28.4 / 42.2 / 87.9 tok/s), plus compute-unit costs of 1.54 CU/h for the L4, 5.30 for the A100 and 8.90 for the G4.

generation:

50.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether NInfer can run on a V100 16GB plus RTX 4070 12GB, reporting 50 t/s on llama.cpp and 35 t/s on coding at long context. Setup is llama.cpp with an IQ3_XXS Flash-Next quant; the machine has 32GB of system RAM. The user also gets a sustained 40 t/s on Strata with the same IQ3_XXS Flash-Next, which they find performs better than a Q4 27B, but the 32GB of RAM causes slow prefill and occasional freezing.

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next IQ3_S at 60 t/s generation on an RTX 5060 Ti 16GB, running on Strata with system RAM overflow. Setup uses Q8 KV cache at 260k context, with the model overflowing to system RAM. The user also tested Qwen3.8 27B IQ3_S at 25-30 t/s with 120k context and Qwen3.8 27B IQ4_XS at 25-30 t/s with 80k context (100k without MTP). User recommends Flash-Next for coding and 27B for non-coding and SVG generation. Notes that Q8 KV cache produced better quality than Q4 in lm-eval benchmarks (73.5% vs 70% on 200 gsm8k samples).

generation:

148.9 tokens/s

prompt processing (prefill):

1886.8 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 148.86 tok/s narrative decode and 174.67 tok/s code decode on a dual-GPU rig (RTX 3090 Ti + RTX 3090). Setup is Strata with IQ3_S quant, int8 KV cache, 262,144-token context, MTP speculation, --spec 4, --kv-resident 32768, --mmap-experts, vision on, automatic layer split. The 148.86/174.67 figures are the ISO18 profile (PREFILL_HELP), which also gives 1,886.76 tok/s at 10K prefill and 3,641.54 tok/s at 90K. ISO19 (PF_FUSED) is the prefill winner at 1,995.53 tok/s at 10K and 4,149.45 tok/s at 90K, but about 5% slower on narrative decode and 3% slower on code decode; its output-sanity check was still pending. The post traces a Strata 0.1.41 regression (decode about 8% slower, 10K prefill about 15% faster, 90K prefill about 7% slower) to automatic GPU placement, and verifies a fix (STRATA_SPLIT_UNIFORM=1 plus gpu_order: as_given) that restores the 0.1.40.3 baseline range. The fix is opt-in and validated on this specific rig.

generation:

18.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Swift-1.5-Qwen3.8-Flash-Next at IQ2_XS on an RTX 3060 Ti 8GB with 32GB system RAM, getting 18 t/s generation at 128k context. Setup is Strata with IQ2_XS quant, int8 KV cache, and expert offloading to system RAM. Prefill is 27 t/s with MTP enabled (334 experts cached in GPU) and 33 t/s with MTP off (1027 experts cached in GPU). User is happy with generation speed given the low VRAM but is struggling with prefill performance and is asking for suggestions.

generation:

44.5 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next IQ3_S at 44.5 token-weighted t/s on 2x Quadro RTX 5000 with 121 GiB RAM. Setup is Strata with IQ3_S GGUF (79 GB), KV int8, 131072 context, spec 4 MTP speculative decoding, resident-experts mode. User ranks this the best all-round agent of the quants tested: 9 of 10 tasks finished RC=0 with verified deliverables, built a full board game overnight. Per-request decode averaged 49.3 t/s (median 50.4), prefill 250.5 t/s average. During real agentic runs with reasoning on, decode was 33-40 t/s. IQ3_S only fit in full residency on the 121 GiB machine; it did not fit on the 62 GB or 30 GB machines.

generation:

15.0 tokens/s

prompt processing (prefill):

100.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 15 t/s generation and about 100 t/s prefill on an M4 Max 36GB MacBook, running a 256K-token prompt. Setup is Inco Splash; no quant or KV cache format stated. User questions how the run fits in 36GB of RAM without significant swap.

generation:

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next IQ2_XS at 147 tok/s decode on an RTX 5070 Ti 16 GB with 64 GB DDR5-4800, after running Strata's --calibrate. Setup is Strata v0.1.43 with IQ2_XS GGUF, int8 KV cache, 131k context, --kv-resident 32768, MTP --spec 4, reasoning_budget_tokens 8192, conversation cache 4 GiB, vision encoder on CPU. Calibration raised median decode from 118 to 143-147 tok/s and cut the full 28-task x2 run from 20.6 to 15.4-16.0 min, but the benchmark score dropped from 50 to 44/56 twice, all in soft exact-match cases. Greedy decoding scored 50/56 versus 37 with Qwen's recommended sampling (temp 1.0/top_p 0.95/top_k 20), 43 at temp 0.6 and 40 at temp 1.0 + min_p 0.05. reasoning_effort high scored 42/56 and was twice as slow. Run-to-run noise is about +/-3/56. Real agent workload (Hermes, 54k context) gave about 100-104 tok/s decode with 78-83% expert-cache hit rate before calibration. RAM runs at 4800 MT/s with EXPO off, which the user estimates costs a few to ~20% versus DDR5-6000.

generation:

33.3 tokens/s

prompt processing (prefill):

406.9 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Q4_0 at 33.28 t/s decode and 406.9 t/s prefill on 2x Tesla P100 16GB with patched llama.cpp, up from 23.43 t/s decode and 233.6 t/s prefill on stock. Setup is llama.cpp v0.7.0 with 41 sm_60 patches, tensor split across two P100s, 5 interleaved rounds per config. The patch set also gives 77.6 t/s decode on Ornith-1.0-35B-A3B UD-Q4_K_M with tensor split, and layer split gives 20.68 t/s decode on the 27B. Tested only on 2x P100; some patches are generic for all cards.

generation:

60.0 tokens/s

quant:

Q4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B MTP Q4 at 60 tokens/s on 2x RTX 3090 with 256GB RAM and an EPYC 7443. Setup is Ollama with shared scheduling across both cards; a single card gave 40 tokens/s. User asks whether this is the sweet spot for the hardware and whether an NVLink bridge would help further.

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 60 t/s average code generation on empty context, with peaks of 60-70 t/s, on two RTX 3060s. Setup is Strata with IQ3_S weights and MTP speculation enabled; the user tuned --pcie-frac to 0.2, --pool-workers, --trim-stage-weights, --pipeline-windows 6, and forced GPU 1 (the faster PCIe lane) as the main GPU via gpu_order as_given. The user says the previous baseline was about 35 t/s decode peak, and that performance degrades with full context but still holds a solid +15-20% over the old baseline.

generation:

53.1 tokens/s

quant:

FP8 (FP8)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 53.1 tok/s single-stream on four Radeon AI PRO R9700 32GB cards in one box. Setup is vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14) with the official FP8 checkpoint, FP8 KV cache, and the model's MTP head with 8 draft tokens, two engines per box at TP=2, 256K max context. At 16 users the box serves 484 tok/s aggregate (about 40 tok/s per user); a random 1024-in/512-out profile gives 623 tok/s aggregate at 837 W. MTP acceptance is 83% on JSON extraction versus 39% on MT-Bench. FP8 scored 98.0/100 versus MXFP4's 95.3/100 on the user's extraction workbench, at 4,477 versus 4,819 validated jobs an hour. Three such boxes were built at about $10,733.91 each.

generation:

100.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at around 100 tokens/s on 2x RTX 3090. Setup is Strata with IQ3_S quant. User is happy with the speed and says quality is a bit better than 27B at Q8. User also owns five more RTX 3090s (one ASUS ROG Strix, four Zotacs) and is deciding whether to upgrade the X299 machine, build a second 4x3090 rig, or sell the extra cards.