Local model performance on your hardware

generation:

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next IQ2_XS at 147 tok/s decode on an RTX 5070 Ti 16 GB with 64 GB DDR5-4800, after running Strata's --calibrate. Setup is Strata v0.1.43 with IQ2_XS GGUF, int8 KV cache, 131k context, --kv-resident 32768, MTP --spec 4, reasoning_budget_tokens 8192, conversation cache 4 GiB, vision encoder on CPU. Calibration raised median decode from 118 to 143-147 tok/s and cut the full 28-task x2 run from 20.6 to 15.4-16.0 min, but the benchmark score dropped from 50 to 44/56 twice, all in soft exact-match cases. Greedy decoding scored 50/56 versus 37 with Qwen's recommended sampling (temp 1.0/top_p 0.95/top_k 20), 43 at temp 0.6 and 40 at temp 1.0 + min_p 0.05. reasoning_effort high scored 42/56 and was twice as slow. Run-to-run noise is about +/-3/56. Real agent workload (Hermes, 54k context) gave about 100-104 tok/s decode with 78-83% expert-cache hit rate before calibration. RAM runs at 4800 MT/s with EXPO off, which the user estimates costs a few to ~20% versus DDR5-6000.

generation:

81.9 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next IQ3_S at 81.9 tok/s in calibration and 85.4 tok/s in chat on an RTX 5070 Ti 16GB with 64GB RAM. Setup is Strata with IQ3_S GGUF, int8 KV cache, 256K context, KV streaming with 32768 resident cells, and MTP speculation. The model is 125B-A6B with 46.84 GiB of experts loaded and an 84GB download. Two concurrent 256K slots exceeded a 58 GiB container limit during startup with and without batch MTP, and earlier unrestricted runs caused global OOMs and a host freeze. The user settled on one active request at 256K with KV streaming.

generation:

53.5 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 53.5 tok/s with a cached prefix on an RTX 5070 Ti 16 GB, up from 17.2 tok/s before tuning. Setup is Strata with IQ3_S at 262K context, KV cache in RAM streaming to a 32K VRAM window, speculative decoding with depth 6 and min-p 0.70, and 13 pool workers on a 14700KF with 96 GB RAM. Cold run was 43 tok/s; the 53.5 figure is with the prefix cached, which the user says is what agent sessions actually do. The user also reports the calibrator's own curve at 42.1 tok/s at --pcie-frac 0.0 down to 20.7 at 0.75.