Local model performance on your hardware

generation:

148.9 tokens/s

prompt processing (prefill):

1886.8 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

mtp (multi-token prediction):

on

User reports Qwen3.8-Flash-Next at 148.86 tok/s narrative decode and 174.67 tok/s code decode on a dual-GPU rig (RTX 3090 Ti + RTX 3090). Setup is Strata with IQ3_S quant, int8 KV cache, 262,144-token context, MTP speculation, --spec 4, --kv-resident 32768, --mmap-experts, vision on, automatic layer split. The 148.86/174.67 figures are the ISO18 profile (PREFILL_HELP), which also gives 1,886.76 tok/s at 10K prefill and 3,641.54 tok/s at 90K. ISO19 (PF_FUSED) is the prefill winner at 1,995.53 tok/s at 10K and 4,149.45 tok/s at 90K, but about 5% slower on narrative decode and 3% slower on code decode; its output-sanity check was still pending. The post traces a Strata 0.1.41 regression (decode about 8% slower, 10K prefill about 15% faster, 90K prefill about 7% slower) to automatic GPU placement, and verifies a fix (STRATA_SPLIT_UNIFORM=1 plus gpu_order: as_given) that restores the 0.1.40.3 baseline range. The fix is opt-in and validated on this specific rig.

generation:

105.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

IQ3-XXS (GGUF)

kv:

int8

User reports Qwen3.8 Flash-Next Swift-1.5 at 105 t/s decode and around 2000 t/s prefill on an RTX 3090 Ti plus a V100 16GB. Setup is Strata 0.1.39 with IQ3-XXS weights, int8 KV cache, 393216 max context, yarn rope scaling at 1.5, and speculative decoding with --spec 4. Layers are split 2/3 on the 3090 Ti and 1/3 on the V100. User says the same model previously reached only around 50 t/s decode with llama.cpp layer splitting.