Gemma 4 12B
on NVIDIA RTX 5080 · llama.cpp · 8,192 ctx
Use cases
long-contextagentic
Summary
User reports Gemma 4 12B QAT at about 78 tok/s decode and 3,100 tok/s prefill on an RTX 5080, at 8K-token prompts.
Setup is a pinned llama.cpp CUDA build with Q4_0 weights, 128K default context (up to 256K) and an FP16 or Q8_0 KV cache on the 16 GB card.
In the user's memory-pipeline tests, saved sessions rose from 52/96 to 94/96 and stale or forbidden claims fell from 36 to 2; on the same pipeline Gemma kept 120-124 correct current facts against Sonnet 5.5's 96 of 135, though Sonnet made 1 error where Gemma made 12-14. Gemma 4 26B-A4B needed changes to fit on 16 GB and did not reduce errors.