Gemma 4 31B
on NVIDIA RTX 4090 · Glyd · 32,768 ctx
Summary
User reports Gemma 4 31B QAT at 46 tok/s on an RTX 4090.
Setup is Glyd 0.31 with Google's QAT 4-bit checkpoint (4.5 bits per weight) at 32K context, using 21.8 of 24.6 GB.
With Google's draft model, chat prompts reach 78-80 tok/s and predictable text up to 176 tok/s. A 30K-token prompt took 20 s. On an RTX 3090 the same model runs at 26 tok/s; a 12B QAT on an RTX 4080 SUPER runs at 82.5 tok/s plain and 109 tok/s with the draft model. Quality vs Google's QAT weights: mean KLD 0.018, median 0.003, top-1 96%, GSM8K 72/75, MMLU-Pro 70/75.