Local model performance on your hardware

generation:

170.0 tokens/s

quant:

QAT

User reports Gemma 4 26B-A4B QAT at about 170 tokens a second on one RTX 4090, with a full two-call pass taking 5 seconds. Setup is Ollama with QAT weights, one card, 96 GB system RAM; vLLM was faster at 3.5 seconds per pass but held 94% of the card against Ollama's 70%. Sending both calls of a pass together cut it to 3.8 seconds on Ollama and 2.8 seconds on vLLM. MedGemma 27B Q4_K_M took 18 seconds per pass. The user asks about speculative decoding, other engines, other models, and CPU offload.

generation:

149.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports about 150 t/s generation. The model is described as a star performer.