Local model performance on your hardware

generation:

64.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks 18 tool-calling models on 33 recorded DevOps failure scenarios on a single RTX 5090 32GB, reporting gemma4:31b at 64 tokens/s with a 94% pass rate and 23.1 GiB measured VRAM. Setup is Ollama 0.35.0 with each model's default build (Q4_K_M, except gpt-oss:20b at MXFP4), 32k context, temperature 0, one run each; VRAM is what nvidia-smi showed for the Ollama process. The top six models tie between 88% and 94%, separated by speed and memory; the fastest model on the card (nemotron-3.5-lightning:30b at 427 tokens/s) passes 52%, and gemma4:12b passes 88% in 9.2 GiB. Ollama's own ollama ps size under-reported real GPU usage by 0.6 to 4.2 GiB for 17 of 18 models. north-mini-code-1.0 could not run due to a CUDA illegal memory access error. Nothing spilled to CPU.

generation:

15.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 15 tok/s on an RTX 5090. Setup is Ollama with Row-Bot 5.0, running locally with no API keys or cloud. The model read a year of bills, a tenancy agreement, an insurance policy and a rent increase letter, charted a CSV, quoted lease clauses, saved 8 linked memories to a local knowledge graph, and drafted an email. Some turns took over 2 minutes.

generation:

171.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B at 171 tok/s on an RTX 5090, solving 14 of 16 hard problems. Setup is Ollama with Q4_K_M GGUF at 64K context, temperature 0.2 and a 32,768-token generation cap, using 17.5 GB of VRAM. Nemotron 3.5 Lightning reached 216 tok/s and 14/16 at 25.4 GB, while Muse Glimmer reached 81 tok/s and 15/16 at 16.9 GB.

User reports inconsistent output from local models Gemma 4 and Qwen 3.6 35B A3B compared to frontier models, for a dictation and SOAP scribe. Setup is a Core Ultra 9 with 128 GB RAM and an RTX 5090, running Proxmox with AnythingLLM and Ollama on llama.cpp. The system prompt is a 25-30k token markdown file.