Measurement
How accurate is our local LLM speed calculator?
By llamaperf · · 6 min read
Quick answer
On dense models, half of the calculator's speed estimates land within about 20% of what people measure, and they don't lean fast or slow. Mixture-of-experts models miss by more, around 45%, because their speed depends on the architecture as much as on memory bandwidth. Use it to build a shortlist, then check the reports for your card before you download 20 GB.
How we checked the estimate
Same card, same model, same quant level
The calculator predicts generation speed from memory bandwidth. Every new token means reading the model's active weights from memory once, so no card can go faster than its bandwidth divided by those bytes. We take that ceiling and multiply it by the share of bandwidth real kernels sustain. The engine began as a port of llmfit, and wherever we have nothing better it still runs on llmfit's constants.
For the test, we paired every estimate with what people measured. A pair is one card, one model size and one quant level, a 27B dense model at 4 bits on an RTX 3090 for instance. The measured side is the median of the reports for that combination, and the other side is the calculator's answer for the same thing. Divide the first by the second and you get a ratio. 1.0 is a perfect call, and 1.5 means people got half again the speed we predicted.
Only runs the estimate is meant to describe
The calculator estimates one request on one device, with no speculative decoding and the whole model in the device's memory, so only reports like that count. Keep part of the model in system RAM and the CPU and its memory set the pace. Add a draft model or MTP and a run can go several times faster than the card alone. The site shows both kinds separately, and this check leaves them out. The performance guide explains each kind of run.
What the numbers show
Dense models land close
These are the results from 1 October 2026. The median ratio says whether the estimate leans fast or slow. The typical miss is the median distance from a perfect call in either direction, so 21% means half the pairs land within 21%.
| Group | Pairs | Median measured / estimate | Typical miss | Within 2x |
|---|---|---|---|---|
| Dense models | 62 | 0.99 | 21% | 89% |
| Mixture-of-experts models | 42 | 1.13 | 46% | 86% |
| Discrete GPUs | 63 | 1.03 | 29% | 84% |
| Macs | 27 | 0.89 | 30% | 89% |
| Strix Halo, DGX Spark, Jetson | 14 | 1.29 | 40% | 100% |
Dense models have a median ratio of 0.99, with no overall bias and half the estimates within 21%. Macs run about 10% slower than predicted, even though each chip's efficiency comes from its own llama-bench run.
Most pairs rest on one report
Any single report brings its own noise, whether that's a different engine build or a laptop running hot. The llama.cpp maintainers say as much about their own benchmark threads:
"Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same." (llama.cpp: Performance of llama.cpp on Nvidia CUDA)
Some of every miss in the table comes from the reports, then. In the few pairs that several people measured, the typical miss falls to 17%.
The miss we fixed
Two machines ran far ahead of the estimate
When we first ran this check, Strix Halo and DGX Spark had a median ratio of 1.80. People were getting almost twice the speed we predicted, and only 64% of their pairs fell within 2x. Both were on llmfit's generic figure of 55% of peak bandwidth, which every card without a benchmark of its own gets.
Measured runs replaced the generic figure
llama.cpp keeps scoreboards where owners time the same 7B model on their own hardware. Strix Halo generates 50.0 tokens per second there (ROCm scoreboard), and DGX Spark 57.2 (CUDA scoreboard). That's 74% and 79% of their bandwidth, and the calculator now uses those figures, which brought their median ratio down to 1.29 with every pair inside 2x. The CUDA thread warns about this kind of machine itself: "For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!" What's left of their gap comes from mixture-of-experts models.
Why mixture-of-experts models miss more
The architecture sets the speed
For each token a mixture-of-experts model reads only its active experts, which puts the bandwidth ceiling very high. Each token also pays for routing and for lots of small expert kernels, though, and how much that costs depends on the model. The calculator covers it with llmfit's flat penalty by expert count. gpt-oss-120B and Qwen3.5-35B-A3B both have more than 64 experts, so the penalty treats them the same, and they don't run alike.
In llama-bench on an RTX 5090, gpt-oss-20B generates 282.5 tokens per second (gpt-oss guide). Qwen3.5-35B-A3B reads less per token by our byte count, yet generates 194.0 on the same card in the same tool (RTX 5090 vs R9700 results), the same 194 a reader reported on llamaperf for that model and card. That gap is the architecture, and a formula tuned on one of these models will misjudge the other.
Why we didn't refit it
We tried a formula fitted to controlled llama-bench runs, bandwidth time plus a fixed per-token cost for each backend. On the runs it was fitted to, the typical miss went from 36% to 11%. On reports from people running the models they use day to day it did worse, predicting speeds about 50% above what they got. A calculator tuned to benchmark models would oversell the ones you'd really run, so the current formula stays until there's enough data per architecture. If you're weighing one of these models, our mixture-of-experts guide covers what they cost in memory against what they give back in speed.
Frequently asked questions
How do I estimate tokens per second for a GPU?
Take the card's memory bandwidth, divide by the bytes of active weights the model reads per token, and multiply by the share of bandwidth real kernels sustain, usually 55% to 80%. A 27B model at 4 bits reads about 15 GB a token, for example, so an RTX 3090 at 936 GB/s tops out near 62 tokens per second, and real runs come in below that.
Is memory bandwidth the bottleneck for LLM inference?
When you're generating tokens for one request, usually yes, because each token reads the active weights from memory once. Prompt processing works on many tokens at once and leans on compute instead.
Why is my tokens per second lower than the calculator says?
Check whether part of the model is sitting in system RAM or your context is much longer than the calculator's default. An old engine build can also explain the gap. The estimate handles mixture-of-experts models poorly. If several people have reported your card and model, trust their median over the estimate.
Why are MoE models slower than their active parameters suggest?
Every token also pays for routing and for many small expert kernels, and that cost varies with the architecture. By our byte count Qwen3.5-35B-A3B reads less per token than gpt-oss-20B, and in llama-bench on an RTX 5090 it still runs about a third slower.
How fast is Strix Halo for local LLMs?
About 50 tokens per second on LLaMA-2 7B at Q4_0, going by llama.cpp's ROCm scoreboard. That's roughly 74% of its 256 GB/s bandwidth, and bigger models slow down in step with the bytes they read per token.