Local model performance on your hardware

generation:

120.1 tokens/s

prompt processing (prefill):

4700-4800 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 120.10 tok/s decode at 32K context on one RTX 4090, with 112.55 tok/s at 128K and roughly 4,700-4,800 tok/s on uncached 32K prefill. Setup is Strata with GSQ IQ3_S GGUF weights, int8 KV cache and a 32K resident window, MTP speculation with four draft tokens, 11 pool workers and a 0.35 PCIe fraction; all experts and the IQ4_NL ngram table stay in RAM, so the run is partially CPU-offloaded. The host is a Ryzen 7900 with 192 GB RAM at a 280 W GPU limit. The 120.10 tok/s figure is the median of launch medians for the selected 24-expert-cache profile, a +0.88% change over the 96-swap baseline; the 128K figure is +4.26%. Earlier matched decode cells give 110.8 tok/s median for the text profile and 97.65 tok/s for the GPU vision profile, and a matched A/B isolates GPU vision residency at 114.85 versus 104.25 tok/s (-9.2%). The 397B partial-offload probe reached 12.42 tok/s with quality unqualified.

generation:

106.1 tokens/s

prompt processing (prefill):

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 106.1 tok/s whole-request on an RTX 4090 24GB. Setup is Strata with IQ2_XS, 128K context, 8-bit KV cache, MTP on, KV streaming on, images on, speed projection off. The IQ3_XXS tier ran at 98.1 tok/s whole-request with ~120 tok/s peak decode and 131 tok/s prefill; the user notes it was 3.7x longer and qualitatively deeper for only 8% less speed. Both tiers match the README's 24 GB card estimate of 100-140 t/s.

generation:

166.4 tokens/s

quant:

IQ2_XS (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 166.36 output tokens/s on a rented RTX 4090 24 GB. Setup is Strata with IQ2_XS, INT8 KV cache, 65,536-token context and 32,768 GPU-resident tokens, MTP draft 4 at 0.5 minimum probability. The rate is a weighted generation figure (sum of completion tokens over predicted time), not whole-request throughput; MTP acceptance was 84.9% and peak sampled GPU memory 23,918 MiB. The generated Hill Climb Racing-style game was rejected for poor vehicle physics and game feel.