Local model performance on your hardware

reported speed:

43.9 tokens/s generation · 1817.0 tokens/s prompt processing

quant:

Q6_K_XL (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 43.9 tok/s generation and 1817 tok/s prompt processing on an RTX 3090 + RTX A4000 at 32k context. Setup is Strata with UD-Q6_K_XL GGUF and int8 KV cache, 262144 native context, speculative decoding with MTP, experts offloaded to CPU. At 200k context Strata Q6 does 48.8 tok/s versus llama.cpp's 10.6 tok/s, and TTFT drops from 595 s to 118 s. Q6 is 12-31% faster than Q8 and 19 GB smaller.

reported speed:

97.0 tokens/s generation

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 97 t/s decode on 2x RTX 3090 with Strata, with about half the experts in system RAM. Setup is Strata v0.1.40.1 with unsloth UD-Q4_K_XL GGUF, a layer split, speculative decoding, and 121 GB DDR4 on a Ryzen 9 3950X. The 97 t/s is at a 95% expert hit rate; shrinking the cache to 90% and 81% hit rates gave 84 t/s and 68 t/s. The user estimates +34% decode from eliminating misses on their box and about +75% headroom at a single-24GB-card share, and reports +16% decode from --pcie-frac 0.2 --spec-min-p 0.7 over about 280 runs. The prefetch feature is not implemented yet.

reported speed:

19.8 tokens/s generation · 437.8 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports GLM-5.3-Flash at 19.8 tok/s decode and 437.8 tok/s prefill on one RTX 3090 24 GB with 251 GB RAM. Setup is the Strata engine with a UD-Q4_K_XL GGUF pack, --chunk 4096, 16K-token prompt, tiered expert cache streaming from RAM and NVMe. Decode splits into 12.4 ms expert copies, 0.2 ms expert kernels and 37.2 ms dense/sync per 49.8 ms token. A 64K prompt prefills at 420.7 tok/s; a 1K prompt at 132 tok/s prefill and 19.8 tok/s decode; disk-only tier decodes at 1.15 tok/s. --chunk 8192 OOMs on 24 GB.

reported speed:

116.0 tokens/s generation · 2908.0 tokens/s prompt processing

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 116 t/s decode and 2,908 t/s prompt read on two RTX 3090s with NVLink. Setup is a modified Strata build (v0.1.38 plus 29 commits) with IQ3_S GGUF at 262K context, greedy, median of 3, on a Ryzen 9 3950X with 121 GB RAM. The second card acts as a peer expert tier read over NVLink; the pair holds about 19,500 of 24,576 IQ3_S experts with a 0.98 hit rate on real agent chats. Decode gains from the second card are modest (+10%) because the verify window is latency-bound on the primary. Sampled with official settings it sits around 102-104 t/s, a 1.5K prompt reads at ~1,670 t/s, and switching chats takes 1 s. One stream at a time; answers are word-for-word identical to the one-card run.

reported speed:

37-40 tokens/s generation · 1700.0 tokens/s prompt processing

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports running DeepSeek V4 Flash with Strata on an RTX 3090 alone at 1700 t/s prefill and 37-40 t/s generation, and on both an RTX 3090 and RTX 5070 Ti at 1850 t/s prefill and 45-50 t/s generation, at 131K context. Setup is Strata with UD-Q4_K_XL quant, 96GB DDR4 system RAM, RTX 5070 Ti on PCIe x16 and RTX 3090 on PCIe x4. User compares against Unsloth Studio on the same model at 131K context, which gave 100 t/s prefill and 14-16 t/s generation, with a 100K conversation taking over 10 minutes to process. User notes Strata initially lacked prefix caching but now supports it, and that the same generation speed matches Qwen3.8-27B-UD-Q8_K_L on both GPUs.

reported speed:

34.0 tokens/s generation · 520.0 tokens/s prompt processing

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at Q8 running at 34 t/s generation and up to 520 t/s prefill on 2x RTX 3090 with 256GB of system RAM. Setup uses the Strata engine with MTP set to 5; most of the model sits in system RAM rather than GPU memory. For comparison, the user says llama.cpp without MTP gives 13 t/s generation and 115 t/s prefill on the same setup, and notes MTP usually does not help when most of the model is in system RAM.

reported speed:

80-110 tokens/s generation

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-110 tok/s on 2x RTX 3090 with 96GB DDR5 system RAM. Setup uses the Strata engine with a Q4_K_XL GGUF quant from bitlamas, at 2500 prompt tokens. The user calls it the biggest thing since Qwen 3.8 27B, offering the speed of Qwen 3.5 35B-A3B with higher intelligence than 3.8 27B.

reported speed:

167.0 tokens/s generation · 2466.0 tokens/s prompt processing

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 2466 t/s prompt processing and 167 t/s generation on an RTX 3090 plus RTX 5070 Ti with the Strata engine and IQ3_S quant. Setup is Strata with IQ3_S; the user also added UD-Q4_K_XL support, which reached 2341 t/s prompt and 126 t/s generation. The user doubled Strata throughput over a week of profiling and reached 5x llama.cpp on UD-Q4_K_XL, where the starting point was 6 t/s. Earlier steps were 21 t/s, 27 t/s and 51 t/s on IQ3_XXS.

reported speed:

38-61 tokens/s generation · 1650.0 tokens/s prompt processing

quant:

UD-Q3_K_XL (GGUF)

kv:

fp16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next running on an RTX 3090 24GB with 128GB system RAM, reaching about 1650 t/s prompt processing and 38 to 61 t/s generation depending on context under Strata. Setup is Strata with Unsloth UD-Q3_K_XL quant, fp16 KV cache, 256k context, speculative decoding with 4 draft tokens, and an expert cache of 6517 slots (~14GB). The same model on llama.cpp master gave up to 700 t/s PP and 23 t/s TG. Generation is a range: 38 t/s at 182k context and about 61 t/s at short context. The user notes the quant Strata recommends by default was faster but produced minor errors and lower quality, so Unsloth's was chosen for accuracy.