Local model performance on your hardware

generation:

51.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

8bit

User reports Qwen3.8 Flash-Next at 51 t/s generation and 1500 t/s prompt processing on an RTX 5070 Ti Laptop 12GB with 64GB RAM. Setup is the Strata engine with an IQ3_XXS GGUF quant and 8-bit KV cache, using 131k context. The model was partially offloaded, using 11GB VRAM and 56GB system RAM. The generation speed was measured at 43k context depth. The user compares this to stock llama.cpp, which reached 23 t/s generation and 100 t/s prompt processing with the same quant.

generation:

9-11 tokens/s

prompt processing (prefill):

400-550 tokens/s

quant:

UD_Q4_K_XL (GGUF)

kv:

q8_0

User reports Qwen 3.8 27B at ~9-11 t/s decode and 400-550 t/s prefill on an RTX 5070 Ti Laptop 12GB. Setup is llama.cpp with UD_Q4_K_XL quant, q8_0 KV cache, 98304 context, draft-mtp speculative decoding, and FFN weights offloaded to CPU. The user notes reasoning verbosity blows past 100K tokens in planning, and with Magic Context the session scaled past 3.7M total processed tokens.

generation:

35.0 tokens/s

prompt processing (prefill):

475.0 tokens/s

quant:

IQ1_S (GGUF)

User reports Qwen3.8 Flash-Next at ~35 t/s decode and ~475 t/s prefill on an RTX 5070 Ti Laptop 12GB with 60 GB system RAM. Setup is the Flyweight engine with an IQ1_S GGUF, running MoE experts on CPU or split with a hot set cached on the GPU. Other measurements on the same laptop: Qwen3.8 27B IQ2_XXS at ~40 t/s, DeepSeek-V4-Flash at 6-7 t/s, and Z-Image 1024x1024 in ~15 s with a 35B model loaded alongside.

generation:

4.5 tokens/s

prompt processing (prefill):

20-27 tokens/s

quant:

UD-Q4_K_XL

kv:

Q8

mtp (multi-token prediction):

on

User reports Qwen3.8-27B at around 4.5 tok/s on an RTX 5070 Ti Laptop GPU with 12GB VRAM and 32GB system RAM. Setup is Unsloth Studio with UD-Q4_K_XL (roughly 17.9GB), 8K context, Q8 KV cache, flash attention, one parallel slot and MTP speculative decoding with draft max 2. Around 10,978 MiB stays in VRAM and the rest is offloaded to the CPU. A longer response ran at 3.26 tok/s, a short factual one at 4.42 tok/s and a Python coding one at 4.53 tok/s, with prompt processing around 20-27 tok/s. Draft-token acceptance was around 78-83%.

generation:

59.0 tokens/s

prompt processing (prefill):

155.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User compares Qwen3.8 27B dense at Q2 and Q3 against Qwen3.6 35B-A3B MoE on 12 GB of VRAM. The MoE was fastest and passed a sanity test, while dense Q3 ran at 7.5-9.1 t/s. The user prefers the MoE for local use.

generation:

5.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

User reports Qwen 3.8 27B UD Q4_K_XL on an RTX 5070 Ti Mobile 12GB with llama.cpp, at roughly 1.5-5 t/s in a context-prioritized configuration and roughly 9-11.5 t/s in a speed-prioritized one. Setup uses a q8_0 KV cache, with context at 262144 in the first configuration and 98304 in the second. User praises the model's capability and instruction following.