Local model performance on your hardware

generation:

15-30 tokens/s

quant:

Q6_K (GGUF)

kv:

Q8/Q8

User asks whether to switch from a Qwen3.8 27B Swift-1.5 Q6_K setup to a Qwen3.8 Flash-Next Swift-1.5 IQ3_XXS GSQ RCO quant for agentic work. Current setup runs at 262144 context with Q8/Q8 KV cache on 40GB VRAM across two Radeon VIIs plus an RTX 3080 Max-Q, with 64GB DDR4 RAM, using ROCm and Vulkan. It achieves around 450 t/s prefill at 16384 tokens and 15-30 t/s output with dflash2 speculation. User is unsure whether the lower-quant Flash-Next would be smarter at comparable speeds, and notes that a mixed Vulkan/ROCm backend rules out engines like Strata, likely requiring a custom llama.cpp fork.

generation:

28.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

flash attention:

on

User reports Ornith 1.5 35B-A3B at 28.6 t/s on an RTX 3080 Laptop 8GB with 32GB system RAM. Setup is llama.cpp with a Q4_K_M GGUF, 65,536 context, q8_0 KV cache, flash attention on, all layers on GPU, and 36 MoE layers on CPU. The user finds the model overthinks, struggles to choose an approach, and recovers poorly from errors, though it can execute Unity tasks when explicitly told which tool to call.