llamaperf

Gemma 4 12B

on NVIDIA RTX 5080 · llama.cpp · 8,192 ctx

Tone: positiveNVIDIA hardware
Nov 19, 2026
Throughput
78.0 t/s gen · 3100.0 tokens/s prompt processing (prefill)Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
Quant
Q4_0 (GGUF)
VRAM reported
16 GB

Use cases

long-contextagentic

Summary

User reports Gemma 4 12B QAT at about 78 tok/s decode and 3,100 tok/s prefill on an RTX 5080, at 8K-token prompts. Setup is a pinned llama.cpp CUDA build with Q4_0 weights, 128K default context (up to 256K) and an FP16 or Q8_0 KV cache on the 16 GB card. In the user's memory-pipeline tests, saved sessions rose from 52/96 to 94/96 and stale or forbidden claims fell from 36 to 2; on the same pipeline Gemma kept 120-124 correct current facts against Sonnet 5.5's 96 of 135, though Sonnet made 1 error where Gemma made 12-14. Gemma 4 26B-A4B needed changes to fit on 16 GB and did not reduce errors.