Local model performance on your hardware

generation:

162.9 tokens/s

quant:

NVFP4 GPTQ (GPTQ)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-Flash-Next NVFP4 GPTQ against Unsloth UD-Q4_K_XL on an RTX 5090 32GB with 128GB DDR5-6000 RAM and an Intel Core Ultra 7 270K Plus, running native Strata inference on Windows. Setup is Strata 0.1.42-nvfp4.2 with 128K context capacity, INT8 KV cache, MTP 6, 15 CPU workers and a 1600MiB VRAM reserve. NVFP4 reached 162.9 t/s at 4K, 161.3 t/s at 32K and 154.1 t/s at 128K; Unsloth UD-Q4_K_XL reached 130.5, 131.0 and 127.6 t/s respectively. Time to first visible answer was 1.42/3.90/14.90 seconds for NVFP4 and 1.99/5.24/20.41 seconds for Unsloth. Each figure is the median of three runs on identical synthetic code-review prompts with 256 output tokens, temperature 0 and reasoning disabled. NVFP4 figures come from its best complete session of four, which the user notes favours NVFP4. In quality testing Unsloth passed 73/82 selected tasks versus NVFP4's 70/82, mainly through instruction following; maths and coding tied.

generation:

161.3 tokens/s

prompt processing (prefill):

5537.8 tokens/s

quant:

IQ3_S (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next GSQ-RCO IQ3_S at a median 161.3 tok/s generation on an RTX 5090 with 128GB DDR5-6000, measured on three 240,000-token inputs. Setup is Strata 0.1.43.1 with INT8 KV cache, 32,768 resident KV cells, spec 4, draft floor 0.50, 12 pool workers, a 55 GiB resident-expert budget and a 700 MiB VRAM reserve, under a 262,144-token context ceiling. Median prompt processing was 5537.8 tok/s, median first visible answer 43.475 s and median total request time 45.071 s; the generation range was 159.6-203.4 tok/s. The same nine requests at a 131K ceiling gave 173.5, 181.2 and 183.0 tok/s at 4,096, 32,768 and 128,000 input tokens, and the 262K ceiling gave 171.2, 175.0 and 179.2 tok/s. Unsloth UD-IQ4_XS reached a 121.2 tok/s short-task median after tuning, against 105.0 before, with 3/3 exact recall at 100,510 and 128,510 tokens. The user notes the RAM upgrade comparison is not a RAM-only A/B test and that no statistical significance was established.

generation:

193.8 tokens/s

prompt processing (prefill):

4627.2 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next IQ3_XXS at 193.8 t/s generation and 4627.2 t/s prompt processing on an RTX 5090 with 128GB system RAM. Setup is Strata 0.1.40 with INT8 KV cache, 4096-token context, MTP speculation with --spec 4, and CPU expert-pool offload. The IQ3_XXS GGUF is 75.84 GB with 22.34 GiB of GPU expert cache. User compares three quants: IQ3_XXS (193.8 t/s at 4K), IQ3_S (161.0 t/s), and UD-IQ4_XS (116.3 t/s). Accuracy screening on 14 MMLU-Pro questions scored 12/14 for all three. IQ3_XXS had the highest median generation speed in the 4K test.

prompt processing (prefill):

1796.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 1,796 tok/s prompt and 128-134 tok/s decode on an RTX 5090 32 GB plus an RTX 4070 Ti SUPER 16 GB. Setup is Strata with IQ3_XXS GGUF, int8 KV cache, 32K cells resident per layer, MTP spec 4, vision on, and 262,144 context; the 76 GB model runs with 31 GB of system RAM and experts streamed from an mmap'd GGUF. Decode is a range of 128-134 tok/s median on real sampling and 95-108 greedy; stock 0.1.33 with a layer split at 36 gave 890 tok/s on an 80K prompt and about 110 tok/s decode. Three patches cut major faults from 24 million to 14 thousand and chat switching from 20-48 s to 0.6-1.2 s.