Gemma 4 26B (4B active)
on NVIDIA RTX 5080 · llama.cpp
Summary
User reports a pruned one-subject Gemma 4 26B-A4B pack at 225 tok/s on an RTX 5080, against 223 tok/s for the full model.
Setup is unmodified llama.cpp (b12549) with Q3_K_M GGUF; the pack keeps 64 of 128 experts per layer and is 7.4 GB versus 13.3 GB for the full model. The full model at Q2_K produced broken output.
On a Ryzen 7 9700X with no GPU the pack runs 20.5 tok/s and the full model 20.0 tok/s. Against the full model on 300 blind-judged questions a pack wins or ties 53% at 64 experts per layer and 77% at 96, and is much weaker outside its subject.