Local model performance on your hardware

generation:

34.4 tokens/s

prompt processing (prefill):

215.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User compares Ollama/ROCm and llama.cpp/Vulkan on an RX 7900 XTX. Ollama reaches 215.8 t/s prompt processing and 34.4 t/s generation at 64K context. llama.cpp/Vulkan reaches 192.0 t/s prompt and 35.8 t/s generation at 64K context, and 230.5 t/s prompt and 36.0 t/s generation at 8K context. The user notes Vulkan is only about 4% faster for generation, while Ollama is better at prompt processing at 64K. The user wonders whether higher reported speeds of 60-100 t/s come from MTP/speculative decoding or other optimizations.

generation:

35-45 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E4B at 35-45 t/s on an RX 7900 XTX. Setup is Ollama with the Q4_K_M quant. User calls it the best consumer AMD option.