Local model performance on your hardware

generation:

81.5 tokens/s

quant:

IQ3_S (GGUF)

User benchmarks nine local Qwen 3.8 configurations against Claude Opus 4.6, with Flash-Next IQ3_S split across both GPUs reaching 81.5 t/s decode at 96K context and 75.5 t/s on a short code prompt. Setup is Strata with IQ3_S GGUF weights (83.6 GB) on an RTX 5060 Ti 16GB over PCIe x16 plus an RTX PRO 4000 Blackwell 24GB over PCIe x1, with experts streamed over PCIe; the 27B IQ3_S runs at 93.2 t/s on a short prompt and 48.5 t/s at 32K. The narrow x1 link is a major bottleneck: Flash-Next read a 4K prompt at about 145 t/s on the PRO 4000 alone versus 1,433 t/s on the 5060 Ti. Best local official-sampling HumanEval was 96.3% against 100% for Opus, while local models won MATH-500 (96.5% vs 95.0%) and instruction following (100% vs 94.9%).

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q8

User reports Qwen3.8-Flash-Next IQ3_S at 60 t/s generation on an RTX 5060 Ti 16GB, running on Strata with system RAM overflow. Setup uses Q8 KV cache at 260k context, with the model overflowing to system RAM. The user also tested Qwen3.8 27B IQ3_S at 25-30 t/s with 120k context and Qwen3.8 27B IQ4_XS at 25-30 t/s with 80k context (100k without MTP). User recommends Flash-Next for coding and 27B for non-coding and SVG generation. Notes that Q8 KV cache produced better quality than Q4 in lm-eval benchmarks (73.5% vs 70% on 200 gsm8k samples).

generation:

55.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

IQ1_M (GGUF)

User reports Qwen3.8 Flash-Next Coder at 55 tok/s generation and 1500 tok/s prompt processing on an RTX 5060 Ti 16GB with 32GB system RAM. Setup is Strata with IQ1_M quant at 65k context. A one-shot landing page demo ran at 46 tok/s over about 2 minutes; an interactive 3D globe took 8 minutes plus a minute of JS fixes. User says the IQ1_M output quality surprised them, comparable to above Q3 for other local models.

generation:

94-99 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next at 94-99 t/s sustained decode on 4x RTX 5060 Ti 16GB, with short turns reaching ~110 t/s. Setup is a custom Strata fork (1 main + 3 expert tiers) with UD-Q4_K_XL quant, 524k context via ported rope-scaling, and a draft vocab fix. Prefill measured at 2159 t/s at 16k and 2420 t/s at 220k; 312k warm replay in 12ms. User abandoned the setup because the Q4_K_XL quant loops in agentic use, repeating thoughts and re-firing tool calls, unlike their int4 W4A16 lane. They compare it favorably to their vLLM lane (~40 t/s at 455k with batch 4) but say the quality bar is not met for daily driving.