Reading benchmarks
Why the same GPU can show very different local LLM speeds
By llamaperf · · 6 min read
Quick answer
A speed result depends on far more than the GPU name. Before you read anything into a gap, match the model build, quantization, engine, GPU count and workload. Then check whether each number times reading the prompt or writing the answer, which engines measure separately, and whether it's a server total that adds up many users at once.
Work out which number you're looking at
When a post reports "40 tokens per second", check whether it means prompt processing or generation. An engine usually reads the prompt far faster than it writes the answer.
Prompt processing and generation are separate numbers
llama.cpp's own benchmark tool reports the two apart. The example table in the llama-bench README has a 7B Qwen2 build at Q4_K_M processing a 512-token prompt at 7,340 tokens per second and generating 128 tokens at 120.6 tokens per second on the same machine. The two stages differ by a factor of about 60 on the same card and file. Put one person's prompt figure next to another person's generation figure and a healthy GPU looks broken.
Servers add a third kind of number
Serving engines split the stages too: vLLM's metrics reference lists prompt tokens, generation tokens and time to first token as separate metrics. A server busy with eight people can also report their combined output, which is far higher than what any one of them sees.
Find out which of these each result measured before you compare them. When a post doesn't say, you can still use the number as an observation, with the context missing. Our performance guide goes through the difference in more detail.
Line the reports up side by side
Put each setup in a small table, one row per field. A lot of disagreements disappear once the rows sit next to each other.
The fields worth comparing
| Field | Why to check it |
|---|---|
| Model and variant | Similar names can be different downloads |
| Quantization | Different weights, sometimes a different code path |
| Engine and version | The software is part of the setup |
| GPU count | A card name doesn't say how many were used |
| Input and output length | The workloads may differ a lot |
| Concurrency | A server total isn't one user's experience |
| Cache and warmup state | Reused work and startup both change timing |
Quantization is the field that catches people out most often. You'd expect the smaller file to run faster, and in the Llama 3.1 8B table in the llama.cpp quantize README it doesn't: the 3.42 GiB IQ3_S build generates 69.31 tokens per second, while the bigger 4.36 GiB Q4_K_S build manages 76.71.
Leave unknowns blank
If the source doesn't give a field, leave it empty. Writing in the usual default makes the table look complete and hides the fact that you guessed. For two cards, the compare page lines up community reports side by side, which is a quick way to spot the fields that differ.
Reproduce the gap yourself
To reproduce the gap, fix the prompt and the output limit first, and save your starting configuration before you touch anything else.
Repeat every run
Run each setup several times and keep every result, the slow ones too. llama-bench does the repeating for you:
"Each test is repeated the number of times given by
-r, and the results are averaged." (llama-bench README)
It runs five repetitions unless you ask for a different count, and the README says the output includes the standard deviation, so you can see how far the runs drifted.
Change one setting at a time
If you swap the model file and the engine in one go and it speeds up, you've found a faster setup without learning why. If all you want is something to use every day, you may not care. Anyone reading your result, though, can't tell which change did it.
Write down whether the model was already loaded and whether you sent the identical prompt twice. Some engines reuse the work from an identical prompt, and the repeat then comes back much quicker than it would for a real user.
Post a number other people can use
What to include
Post the setup with the number, along with how you measured it and whether it's a single run or the median of several. If the results bounced around, give the range instead of your best run.
When you compare two setups, say which fields match and which don't. If you couldn't control the test, it's fair to say you have two community reports with different conditions and stop there. A bigger number on one card doesn't make that GPU faster when everything else differs too.
Where to send it
The report template keeps the fields from the table above in one file. Fill it in and submit your result, slow or fast. Include the settings even when the result is slow, so another reader can reproduce it. If you haven't settled on a model yet, read how to choose a local LLM for your hardware first.
Frequently asked questions
Why is my smaller GGUF getting fewer tokens per second than a bigger one?
Quant types differ in more than size. Each one has its own code path, and some of the small ones decode more slowly. llama.cpp's Llama 3.1 8B table shows exactly this: the 3.42 GiB IQ3_S build generated 69.31 tokens per second and the larger 4.36 GiB Q4_K_S build generated 76.71. Before you blame the quant, though, make sure both builds ran fully on the GPU.
How do I know if my tokens per second is normal for my card?
Look for reports with the same model build, quantization and engine on a single card like yours, and make sure your model is fully on the GPU (in Ollama, ollama ps should say 100% GPU). If all of that matches and you're still well off, suspect drivers, power limits or something running in the background.
Is tokens per second a useful metric at all?
It is, once you know which one you're reading. Output speed is how fast text streams after it starts. Time to first token is how long you stare at a blank screen before that, and with a long prompt it's often the bigger wait.
Why does the same prompt come back so much faster the second time?
Two things help the repeat. The model is already in memory, and Ollama keeps it there for 5 minutes by default, so a second request inside that window skips loading. Some engines also reuse the work from an identical prompt.
Does a server's tokens per second tell me what one user will get?
Usually not. A server figure often sums the output of every active request. What you want is a per-request figure, or time to first token measured with a single user.
Where can I compare my speed with other people on the same GPU?
The llamaperf GPU pages list community reports for each card, with the model, quantization and engine behind each one. Open the original post before you compare, because settings nobody wrote down can still differ.