Local model performance on your hardware

generation:

43.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports glm-4.7-flash at 43.16 tok/s on a Framework host with Core Ultra CPU and RTX 3060 12GB. Setup is Ollama with a 192-token generation cap; the model scored 5/5 on the repo's compact coding, tool-use and agent-orchestration quality suite. It is the fastest perfect-quality model in the capture, ahead of qwen3.5:35b-a3b at 39.90 tok/s and qwen3-coder-next at 32.55 tok/s. A separate llama-server comparison on a T5500 host lists omnicoder:9b-q4_k_m at 1.48 tok/s, qwen3.5:35b-a3b-q2_k_l at 3.28 tok/s, lfm2:24b at 4.36 tok/s and granite4:7b-a1b-h at 6.48 tok/s.

generation:

49.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5 7B Instruct at 49.0 tok/s on an RTX 3060 12GB, and 6.4 tok/s CPU-only on the same machine. Setup is Ollama 0.11.4 with Q4_K_M weights, single-stream, warm speed as the median of three runs with 200-token answers. About 1.2 GB of the card was already in use by the desktop and a Stable Diffusion web UI. The same model measured 48.9 tok/s on the GPU in a separate CPU-vs-GPU comparison, with a cold first answer of 5.6 s on GPU against 31.3 s CPU-only, a 7.6x speed-up. The user notes the CPU is a 2012 Xeon E5-2665 without AVX2, and that a modern CPU with AVX2 or AVX-512 will beat these CPU numbers.

generation:

60.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports E2B at about 60 t/s on an RTX 3060 12GB. The figure is estimated from compute-market tiers rather than measured.

generation:

45.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 45 t/s on an RTX 3060 12GB. The E4B model fits easily.