Local model performance on your hardware

generation:

52.0 tokens/s

prompt processing (prefill):

398.6 tokens/s

quant:

Q2_0 (GGUF)

kv:

int8

User reports Qwen3.8-Flash-Next at 52.0 t/s decode and 398.6 t/s prefill on a single Tesla V100-PCIE-16GB (PCIe Gen3 x16). Setup is the Strata-V100 fork with Q2_0 weights, int8 KV cache, 262,144-token context, MTP with --spec 8 and draft floor 0.70, a 20/28 layer split, and 700 MiB vision reserve; the machine has a Ryzen 5 3600 and 48 GB DDR4-3200. Each value is the median of three fresh uncached requests with exactly 256 output tokens. Decode falls from 52.0 t/s at 1K to 36.8 t/s at 256K; the 128K and 256K prompts hit 84 °C and thermally throttled during prefill. A separate before/after run of the integrated build measured prefill gains of +2.44% at 2K, +3.93% at 8K and +0.94% at 32K.

prompt processing (prefill):

7090.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.8-Flash-Next UD-Q4_K_XL at up to 7,350 tok/s prefill and ~113 tok/s decode on an IBM AC922 with 4x Tesla V100 16GB SXM2 GPUs. Setup is a forked Strata engine with MTP speculative decoding, 72 GiB of experts page-locked in RAM across both sockets, and GPUs pulling from NVLink 2.0 at ~70 GB/s each. The 252K-token prompt prefilled at 7,090 tok/s in 35 s; follow-up at that depth gave first token after 0.26 s and 60 tok/s. Generation varied by workload: ~113 tok/s peak on JSON, ~100 on code, ~84 on prose. Stock llama.cpp on the same machine produced 130 tok/s prefill and 15 tok/s decode.