Local model performance on your hardware

generation:

34.9 tokens/s

prompt processing (prefill):

5334.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Devstral 24B at 34.9 t/s generation and 5334.83 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 4096-token context; the recorded prompt was 2597 tokens, not a full 4096-token prefill. The figure is a median of 3 fresh-prefill runs at temperature 0 and seed 42, with a unique nonce per repetition to avoid prompt-cache hits; the source captured only 4096-context cells and warns against extrapolating to 8K or 16K.

generation:

51.7 tokens/s

prompt processing (prefill):

1928.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3-Coder 30B-A3B at 51.7 t/s generation and 1928.1 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with partial CPU/RAM offload since the model exceeds VRAM. The recorded prompt was 1347 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs; 74% GPU residency was measured separately at 8K context, not this cell.

generation:

93.4 tokens/s

prompt processing (prefill):

4842.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 Coder at 93.4 t/s generation and 4842.2 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on one 16GB card. The recorded prompt was 1368 tokens, not a 16384-token prefill, and the figures are medians of 3 fresh-prefill runs at temperature 0 with a 256-token target.

generation:

171.4 tokens/s

prompt processing (prefill):

9825.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 Coder-7B at 171.4 t/s generation on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on a Ryzen 7 7800X3D under Windows 11. The figure is the median of 3 fresh-prefill runs with a 1368-token prompt, not a 16384-token prefill; prompt processing was 9825.5 t/s.

generation:

176.6 tokens/s

prompt processing (prefill):

9984.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 7B at 176.6 t/s generation and 9984.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, on one 16GB card. The recorded prompt was 1674 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs.

generation:

298.6 tokens/s

prompt processing (prefill):

19397.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 3.2 3B at 298.6 t/s generation and 19397.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, with an actual recorded prompt of 1657 tokens rather than a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, derived from Ollama eval_count/eval_duration rather than TTFT division, with temperature 0 and a unique nonce per repetition to avoid prompt-cache hits.

generation:

94.6 tokens/s

prompt processing (prefill):

4864.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 14B at 94.6 t/s generation and 4864.7 t/s prompt processing on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at a configured 16384-token context; the recorded prompt was 1674 tokens, not a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, with 256 target output tokens and a unique nonce per repetition to avoid prompt-cache hits.

generation:

178.3 tokens/s

prompt processing (prefill):

10328.6 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gpt-oss 21B-A3.6B at 178.3 t/s on an NVIDIA RTX 5080. Setup is Ollama with MXFP4 at 16384 configured context, on one 16GB card; the recorded prompt was 1360 tokens, not a full 16384-token prefill. The figure is a median of 3 fresh-prefill runs at temperature 0 and seed 42, with a unique nonce per repetition to avoid prompt-cache hits; it is the Aug 26 re-capture, not the faster Aug 12 capture.