Local model performance on your hardware

generation:

60-78 tokens/s

quant:

FP4-Local QAD (GGUF)

kv:

INT8

User reports Qwen3.8-Flash-Next FP4-Local at roughly 60-78 tok/s around 40-55k real context on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5. Setup is Strata 0.1.45 with NVIDIA FP4-Local QAD weights, INT8 KV cache at 200k context, a ~38GB RAM expert tier with adaptive VRAM expert cache, SSD-backed PLE and MTP speculative decoding (spec=6). Cold 43k prompt prefill was about 1,890 tok/s with up to ~90% GPU expert-cache hit. BFCL-v4 quick test scored 85% overall and 92% on the 50 single-turn tool-selection tasks.

prompt processing (prefill):

1680.0 tokens/s

quant:

NVFP4 (GGUF)

kv:

INT8

User reports Qwen3.8-Flash-Next NVFP4 running on a single RTX PRO 4500 Blackwell 32GB with 64GB DDR5 system RAM, reaching up to ~80 tok/s at ~50k context and ~60-67 tok/s at ~188k warm context. Setup is the Strata NVFP4 fork with NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding, and W4A8 prefill on Blackwell, served through Strata's OpenAI-compatible server. A cold 189k full prompt measured ~1,680 tok/s prefill and ~53 tok/s decode. User calls the result impressive for a single 32GB GPU with only 64GB system RAM.