Measurement
How to benchmark Ollama generation speed with its API
By llamaperf · · 5 min read
Quick answer
For a finished Ollama request, output speed is eval_count divided by eval_duration converted to seconds. Keep that apart from total request time, and note the model, prompt, context, hardware and whether the model was already loaded.
Save a complete response
With streaming off, Ollama's generate endpoint sends back a single JSON object, and that file is your record of the run.
Send one non-streaming request
Replace the placeholder with the exact name of a model you've installed (ollama list shows them), and keep the prompt and options identical across every run you plan to compare.
curl http://localhost:11434/api/generate \
-H 'Content-Type: application/json' \
-d '{
"model": "YOUR_INSTALLED_MODEL",
"prompt": "Explain how a hash table handles collisions, with an example.",
"stream": false,
"options": {"num_predict": 256}
}' > ollama-run.json
If you only want a quick look and don't need to save anything, ollama run with --verbose prints the same timings after each reply, eval rate included.
Check the response before you do any maths
num_predict caps the output, but the model can stop sooner, so take the token count from the response instead of assuming 256. Look for an error in the file as well. Mark failed requests as failures instead of entering zero for speed.
Work out output speed
You only need eval_count and eval_duration from that file.
The formula
The Ollama API reference defines eval_count as the number of output tokens, and describes eval_duration like this:
"Time spent generating tokens in nanoseconds" (Ollama API reference, eval_duration)
So convert to seconds before you divide:
output tokens/second = eval_count / (eval_duration / 1,000,000,000)
The sample response on that page has an eval_count of 18 and an eval_duration of 52,479,709, which is about 0.05 seconds. Skip the conversion and the speed you publish is off by a factor of a billion.
A worked example
Say a run produced 200 tokens and eval_duration came to 10 seconds. That works out to 20 tokens per second (round numbers, picked to show the arithmetic).
Keep the JSON file next to the result. If all you saved was the final figure, you've no way back to catch a unit slip, or to notice that one run stopped after twelve tokens.
Keep total time separate
The same response also reports total_duration, load_duration and prompt_eval_duration. Keep each field separate to avoid mixing loading time with generation speed.
Loading can take longer than the answer
In the sample response on Ollama's API page, load_duration is 101,397,084 nanoseconds out of a total_duration of 174,560,334, so loading took about 58% of that request. Divide output tokens by total time on a cold start and you'll blame the GPU for time spent reading the file from disk.
The Ollama FAQ sets how long a model stays warm:
"By default models are kept in memory for 5 minutes before being unloaded." (Ollama FAQ)
A test you run six minutes after the previous one is a cold start again.
Time to first token needs streaming
A non-streaming request can't show how long the first word took to appear. For that you need a streaming request and a clear rule for what counts as the first output, and you should write down which rule you used.
If you're picking a model to use every day, time the whole task as well. One that generates quickly but writes three times as much, or needs a follow-up correction, can still take longer to give you an answer you can use.
Repeat it the same way every time
Repeat the test with identical settings. Five runs give you a first view of the variation that a single run hides.
Write the protocol down
Decide up front whether you're timing a cold start or a model that's already loaded, and keep the two in separate groups. Sending the same prompt over and over is fine as long as you note it, since caching can make later runs do less work.
Record the model name, Ollama version, operating system, hardware, context setting and whatever else is running on the machine. A published summary should say what it is (the median of five runs, for example) and list the individual values beside it. Our report template has a field for each of these.
Watch for answers that end early
Read any answer that ends almost immediately before it goes into a comparison. A refusal or an early stop still comes with a timing, and that timing measures the wrong work.
If your eval rate is nowhere near someone else's on the same card, why the same GPU can show very different speeds lists what to line up first, and the benchmark reading guide covers what to record. Once you have clean runs and the JSON files behind them, post the median through the submission form, even if it's slower than the numbers you've seen elsewhere.
Frequently asked questions
How do I see my tokens per second in Ollama?
Add --verbose to ollama run and it prints the eval rate after each reply. If you're calling the API, divide eval_count by eval_duration once you've converted the nanoseconds to seconds.
What's the difference between prompt eval rate and eval rate?
Prompt eval rate is how fast Ollama reads your input, and eval rate is how fast it writes the answer. Reading is usually much quicker, so putting one next to the other doesn't tell you anything useful.
Why is my first request so much slower than the rest?
It's loading the model from disk. After that, Ollama keeps the model in memory for 5 minutes by default, and any request inside that window skips the load.
Should I divide by total duration or by eval duration?
Use eval_duration for output speed. Total duration also counts loading and prompt processing, and on a cold start that makes a perfectly good GPU look slow.
Does num_predict make the model write exactly that many tokens?
No, it's only a cap. The model can stop earlier, so take the real count from eval_count every time.
Should I report the median or the average of my runs?
Use the median for the headline because an unusually slow or fast run affects it less than the average. Put the individual values beside it so people can see the spread.