Qwen3.8 reports

generation:

52.0 tokens/s

prompt processing (prefill):

398.6 tokens/s

quant:

Q2_0 (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 52.0 t/s decode and 398.6 t/s prefill on a single Tesla V100-PCIE-16GB (PCIe Gen3 x16). Setup is the Strata-V100 fork with Q2_0 weights, int8 KV cache, 262,144-token context, MTP with --spec 8 and draft floor 0.70, a 20/28 layer split, and 700 MiB vision reserve; the machine has a Ryzen 5 3600 and 48 GB DDR4-3200. Each value is the median of three fresh uncached requests with exactly 256 output tokens. Decode falls from 52.0 t/s at 1K to 36.8 t/s at 256K; the 128K and 256K prompts hit 84 °C and thermally throttled during prefill. A separate before/after run of the integrated build measured prefill gains of +2.44% at 2K, +3.93% at 8K and +0.94% at 32K.

generation:

80-90 tokens/s

quant:

WNA16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports completing a full 262,144-token sequence with Qwen3.8-Flash-Next on 2× RTX 4090 24 GB, with a boundary run of 261,991 input + 128 output tokens in 123.3 s (≈2,150 input tok/s). Setup is vLLM (v0.5.0 image) with TP2/EP2, 256K context, hot 84 profile, CPU_OFFLOAD_GB=30, MTP_DEPTH=3, and fast-256k switches; peak VRAM was ~23.8 GiB per card. Decode measured 80–90 tok/s (mean ≈86) with the SM 8.9 gate patch, prefill 1,772–1,814 tok/s at ~14K and 2,294–2,316 tok/s at ~61K; the user notes the x8/x8 PCIe bifurcation likely caps stream-staging and that hot 88 OOM'd on an earlier build.

generation:

84.2-88.8 tokens/s

prompt processing (prefill):

3401-3413 tokens/s

quant:

W4A16 (safetensors)

kv:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on two RTX 3090s with 128 GB RAM, reaching 3,401–3,413 tok/s prefill and 84.2–88.8 tok/s decode on a 131,099-token prompt. Setup is vLLM with W4A16 AutoRound weights, BF16 KV cache, FP8 PLE table, MTP3 speculative decoding, and CPU expert offload. The 64 GB RAM profile runs both cards with 64 GB system RAM, keeping 88 experts per layer on each GPU and streaming the rest from RAM; prefill uses 8,192-token chunks. A single RTX 3090 with 64 GB RAM reaches 2,106 tok/s prefill and 47.4 tok/s decode at 131K context.

generation:

120.1 tokens/s

prompt processing (prefill):

4700-4800 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 120.10 tok/s decode at 32K context on one RTX 4090, with 112.55 tok/s at 128K and roughly 4,700-4,800 tok/s on uncached 32K prefill. Setup is Strata with GSQ IQ3_S GGUF weights, int8 KV cache and a 32K resident window, MTP speculation with four draft tokens, 11 pool workers and a 0.35 PCIe fraction; all experts and the IQ4_NL ngram table stay in RAM, so the run is partially CPU-offloaded. The host is a Ryzen 7900 with 192 GB RAM at a 280 W GPU limit. The 120.10 tok/s figure is the median of launch medians for the selected 24-expert-cache profile, a +0.88% change over the 96-swap baseline; the 128K figure is +4.26%. Earlier matched decode cells give 110.8 tok/s median for the text profile and 97.65 tok/s for the GPU vision profile, and a matched A/B isolates GPU vision residency at 114.85 versus 104.25 tok/s (-9.2%). The 397B partial-offload probe reached 12.42 tok/s with quality unqualified.

generation:

106.1 tokens/s

prompt processing (prefill):

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 106.1 tok/s whole-request on an RTX 4090 24GB. Setup is Strata with IQ2_XS, 128K context, 8-bit KV cache, MTP on, KV streaming on, images on, speed projection off. The IQ3_XXS tier ran at 98.1 tok/s whole-request with ~120 tok/s peak decode and 131 tok/s prefill; the user notes it was 3.7x longer and qualitatively deeper for only 8% less speed. Both tiers match the README's 24 GB card estimate of 100-140 t/s.

generation:

166.4 tokens/s

quant:

IQ2_XS (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 166.36 output tokens/s on a rented RTX 4090 24 GB. Setup is Strata with IQ2_XS, INT8 KV cache, 65,536-token context and 32,768 GPU-resident tokens, MTP draft 4 at 0.5 minimum probability. The rate is a weighted generation figure (sum of completion tokens over predicted time), not whole-request throughput; MTP acceptance was 84.9% and peak sampled GPU memory 23,918 MiB. The generated Hill Climb Racing-style game was rejected for poor vehicle physics and game feel.

generation:

90.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Flash-Next at about 90 t/s on a Radeon Pro W7800 48GB with 64GB DDR5 system RAM. Setup is llama.cpp with an IQ3_XXS GGUF quant, 128k context and Q8 KV cache, run with default settings. The user is satisfied with the speed but the model fell into repetition loops after 3-4 minutes of reasoning on a 27KB JavaScript audit task, repeating the same steps dozens of times; they interrupted it twice and ask whether a reasoning budget setting is the cause.

generation:

58.0 tokens/s

prompt processing (prefill):

1563.4 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next GSQ-RCO IQ3_XXS at up to 58 tok/s decode and 1,563.4 tok/s prefill on a 3975wx Threadripper with 256GB DDR4, CPU only. Setup is Strata with 131k context and no speculative decoding. User notes prefill could be tuned higher and MTP/DFlash should be tested, but the IQ3_XXS quant only passed 4/8 quality tests and is not safe as a local agent without supervision.

generation:

36.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~36 tok/s single stream on a single Radeon AI PRO R9700 32GB with 64GB system RAM. Setup is llama.cpp with stew675's RDNA4 patches (r30), AtomicChat Q4_K_M quant, q8_0 KV cache, flash attention, 2 slots x 131k context, -ncmoe 41 and MOE_EXPERT_CACHE_MIB=4096, GPU capped at 220W. Two streams combined reach ~50 tok/s and prompt reading is 526-589 tok/s. The 27B Q5_K_XL is ~1.7x faster per turn and reads prompts 1.6-2x as fast, but scores lower on tool-eval-bench Hard Mode (79.8 vs 93.9).

generation:

37-40 tokens/s

prompt processing (prefill):

1700.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports running DeepSeek V4 Flash with Strata on an RTX 3090 alone at 1700 t/s prefill and 37-40 t/s generation, and on both an RTX 3090 and RTX 5070 Ti at 1850 t/s prefill and 45-50 t/s generation, at 131K context. Setup is Strata with UD-Q4_K_XL quant, 96GB DDR4 system RAM, RTX 5070 Ti on PCIe x16 and RTX 3090 on PCIe x4. User compares against Unsloth Studio on the same model at 131K context, which gave 100 t/s prefill and 14-16 t/s generation, with a 100K conversation taking over 10 minutes to process. User notes Strata initially lacked prefix caching but now supports it, and that the same generation speed matches Qwen3.8-27B-UD-Q8_K_L on both GPUs.

generation:

53.0 tokens/s

prompt processing (prefill):

1620.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 53 tokens/s generation and 1,620 tokens/s prompt processing on an RTX 5070 12GB with 64GB RAM. Setup is the Strata engine with IQ3_S quantization at 128K context; the model's experts are split across GPU, RAM and SSD. The page also lists Q2_0 at 94/2,650, IQ2_XS at 79/2,090, IQ3_XXS at 62/1,750 and Coder at 55/2,180 tokens/s on the same card, and RX 9070 XT 16GB figures of 60/1,160 (Q2_0), 52/1,110 (IQ2_XS) and 44/1,420 (Coder). Unsloth UD-Q4_K_XL is given as 7-8.5 tokens/s on a 64GB PC.

generation:

96.2 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 96.2 tok/s on prose and 141.1 tok/s on code on an RTX 3090. Setup is SGLang with EXL3 3.0 bpw weights and an MTP draft model, single request. The figures come from the trellis-serve project's registry recipes; the same table also lists Qwen3.6-35B-A3B at 251.7 tok/s prose and 357.9 tok/s code on the RTX 3090, and Qwen3.8-27B at 91.2 tok/s prose and 66.1 tok/s code on an Intel Arc Pro B70.

generation:

57.3 tokens/s

prompt processing (prefill):

1166.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares R9V and Strata running Qwen3.8 Flash-Next UD-IQ4_XS on a dual Radeon AI PRO R9700 system with 96GB DDR4 and an AMD 3900X. R9V is 15-18% faster on prefill at every context size, while decode is roughly level between the two engines. Setup is R9V v0.4.4 with UD-IQ4_XS GGUF shards and MTP, single-session requests at 16k, 32k and 64k context with 256 output tokens. Strata v0.1.39 ran on a single GPU because the two-GPU split needed about 118 GB of RAM; its expert cache held 10,774 experts in VRAM with a 98.7% decode cache hit rate. R9V measured 1166 tok/s prefill and 57.3 tok/s decode at 16k, 1156/58.8 at 32k, and 1109/53.8 at 64k. Strata measured 980/53.3 at 16k, 1003/53.8 at 32k, and 986/54.6 at 64k. The host has 92 GiB of RAM against the R9V README's 128 GiB, and the README's claimed 1.6-1.7k prefill was not reached.

generation:

38.1 tokens/s

prompt processing (prefill):

1263.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 38.1 t/s generation and 1263 t/s prefill on 2x RTX 3090, asking whether NVLink would improve performance over a better motherboard. Setup is llama.cpp b10586 (CUDA) behind llama-swap on Windows 11, with a Ryzen 7 5800X3D and 64GB DDR4. Cold load took 24s. At 28k context the run gave 1250 t/s prefill and 36.2 t/s generation; at 85k context 901 t/s prefill and 28.9 t/s generation. User also runs nightly data classification jobs and asks whether prefill speed would benefit.

generation:

72.0 tokens/s

prompt processing (prefill):

1700.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 72 t/s generation and ~1,700 t/s prompt reading on an RTX 4070 Ti Super 16GB. Setup is Strata 0.1.39 with IQ3_S (~84GB) at 256K context, on a Ryzen 9 7950X3D with 128GB DDR5. Out of the box it did 8 t/s; after START-HERE.bat --calibrate it reached 72 t/s. A 17K prompt reads in 8 seconds. User's 27-task coding suite scored 27/27 x3 and HumanEval 159/164. Same model family under llama.cpp with offloading ran at 19 t/s. Keeping the 29GB n-gram table in RAM with --ple-io mmap made results worse (25-26/27 and slower).

generation:

20-25 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether replacing a GTX 1070 with an RTX 3060 12GB is worthwhile for local LLMs and agentic coding. Current setup is an RTX 3080 10GB plus GTX 1070 8GB, Ryzen 5 5600X, 32GB RAM. User reports running Qwen3.8 27B IQ3_XXS at about 20-25 tokens/s with 44K context using Unsloth, and wants at least 128K context. The 3060 upgrade is a purchase question, not a measured run; no throughput figure is reported for the proposed card.

generation:

34.0 tokens/s

prompt processing (prefill):

520.0 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at Q8 running at 34 t/s generation and up to 520 t/s prefill on 2x RTX 3090 with 256GB of system RAM. Setup uses the Strata engine with MTP set to 5; most of the model sits in system RAM rather than GPU memory. For comparison, the user says llama.cpp without MTP gives 13 t/s generation and 115 t/s prefill on the same setup, and notes MTP usually does not help when most of the model is in system RAM.

generation:

110.2 tokens/s

quant:

EXL3 4.05 (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Swift 1.5 (a fine-tune of Qwen3.8-Flash-Next) at ~110.2 tok/s single-stream decode on two DGX Sparks with veloGB10, TP=2. Setup is veloGB10 with an EXL3 4.05 pack (SharkWipf) and MTP speculative decoding; the official doth4580 pack reached 110.6 tok/s and a Huihui abliterated pack 113.6 tok/s on the same hardware. A header-only rewrite of the n-gram shard layout was needed to load community EXL3 packs. User compares against an older vLLM NVFP4 setup at ~52 tok/s (September, not same-day) and reports Swift 1.5 at xhigh effort passing 11/12 hard trap tasks versus 7/12 at medium, at about 2x output tokens and 1.8x wall time. Aggregate throughput at 2-3 concurrent requests is 91-98 tok/s, rising to 157 at 8 and 181 at 16.

prompt processing (prefill):

1680.0 tokens/s

quant:

NVFP4 (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next NVFP4 running on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5 system RAM, reaching up to ~80 tok/s at ~50k context and ~60-67 tok/s at ~188k warm context. Setup is the Strata NVFP4 fork with NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding, and W4A8 prefill on Blackwell, served through Strata's OpenAI-compatible server. A cold 189k full prompt measured ~1,680 tok/s prefill and ~53 tok/s decode. User calls the result impressive for a single 32GB GPU with only 64GB system RAM.

generation:

56.0 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 56.0 tok/s decode and 497 tok/s prefill on 2× RTX 3060 12GB. Setup is llama.cpp CUDA with UD-Q4_K_XL GGUF, q4_0 KV cache, 131072 context, tensor split across the two cards, and MTP speculative decoding. The 106,294-token context gate took 213.8 s. A second recipe runs Qwen3.6-35B-A3B on the same machine at 72.6 tok/s decode and 512 tok/s prefill with q8_0 KV cache and layer split, keeping some expert tensors in system RAM (8.22 GiB host RAM drop).