Gemma 4 31B
on NVIDIA RTX 3060 12GB · Ollama
Summary
User asks about a budget upgrade to run Gemma 4 31B Q5_K_M at 16K context and at least 15 t/s, currently getting 2.3 t/s generation (2.7 t/s with MTP) on an RTX 3060 12GB with 47% GPU / 53% CPU offload.
Setup is Ollama/llama.cpp with Q5_K_M weights and Q8 KV cache on an i5-14400F with 32GB DDR4; the 243-token prompt was processed with the model partly offloaded to system RAM.
User says IQ3_M drops quality without adding speed, and is weighing an RTX 5060 Ti 16GB plus the 3060 on a PCIe 3.0 x1 riser, dual Tesla P100s on an X79/X99 Xeon system, or a cheaper AMD 16GB card.