How to compare local LLM benchmarks

A tokens-per-second number describes a particular run. Before using it to choose a model or buy a GPU, check what was running and how it was measured.

Start with the whole hardware setup

One GPU and four GPUs are different setups, even if the card name is the same. Check GPU count, total memory and system RAM. CPU offloading lets a model use memory outside the GPU, but changes its performance. Some reports describe mixed GPU models in their notes.

Separate generation from prompt processing

Generation speed measures output tokens. Prompt processing measures how quickly the input is processed. Neither number alone tells you how long a complete answer takes. A long prompt can mean a substantial wait before the first output token.

Check how many requests ran at once

Aggregate throughput adds up tokens across concurrent requests. It is useful for serving many users, but it is not the speed one person sees. If a report does not specify concurrency, do not assume its figure is a single conversation.

Match the model and settings

Compare the same model size and quantization. Context length, engine version, KV-cache settings and speculative decoding can also change the result. Missing settings mean the comparison is incomplete.

Distinguish an estimate from a measurement

The VRAM calculator estimates memory use and, where hardware data is available, speed. It also shows community measurements where available. A community report is evidence of one setup, not a guarantee for yours. Open the report and its source to check the conditions.

Measure and share a reproducible run

Choose a prompt representative of your task. Keep the model, settings and hardware fixed; separate loading from warmed-up runs, repeat the workload, and retain every result. Report input caching and background activity. Use the observed token counts rather than the requested output limit.

Ollama: capture one request and its timings

With Ollama running locally and a model already installed, replace YOUR_INSTALLED_MODEL in this request. Save it as request.json, then run the command. Use a new output filename for each repetition.

{
  "model": "YOUR_INSTALLED_MODEL",
  "prompt": "Explain how to compare two local inference runs fairly.",
  "stream": false,
  "options": { "temperature": 0, "num_predict": 128 }
}

curl --fail-with-body http://localhost:11434/api/generate \
  -H 'Content-Type: application/json' \
  --data-binary @request.json -o run-01.json

Generation tokens/s = eval_count × 1,000,000,000 ÷ eval_duration, provided the duration is positive. Preserve prompt_eval_count, prompt_eval_cached_count and prompt_eval_duration separately; cached tokens must not inflate an uncached processing rate. Missing fields remain unknown. This non-streaming request does not measure time to first token.

The request inherits other model/server defaults: record them, including context, thinking and offloading. Sequential requests from this command do not prove the server had no other traffic. See the official Ollama response fields.

llama.cpp: use its dedicated benchmark tool

Use llama-bench with a local GGUF file. Its official instructions describe prompt, generation and combined tests, repetition counts and machine-readable output. Record the exact command and build version. Keep batch size distinct from concurrent user requests; benchmark timings exclude some application overhead, so they are not end-to-end chat latency.

Download the copyable report template →

Fill in the settings you know and retain the raw results. Inspect files for private prompts before sharing. Submit the report with its source; put conditions without a dedicated form field in your notes.

Link to this measurement workflow · Diagnose slow performance

Check models for your hardware →Browse GPU reports →