Keep a local LLM performance log that survives upgrades

Everything you'd need to rebuild it

Start from a configuration that works for you today and save the model name and revision, quantization, engine version and build, hardware, context size and any runtime flags. Keep the command or config file itself too. On llama.cpp, llama-bench takes prompt length, output length and repetitions as flags, which means a rerun months later can match the original exactly. By default it uses a 512-token prompt, 128 generated tokens and 5 repetitions.

"Each test is repeated the number of times given by -r, and the results are averaged." (llama.cpp: llama-bench)

Why bother

Engines move fast, and a speed drop can ride in on an update you barely looked at. In a llama.cpp report from August 2026, a user on an RTX 5080 saw prompt processing on the same first task fall from 1,029 to 413 tokens per second over four nightly builds (llama.cpp issue 26918). They'd logged every build along the way, so they could point to where it went wrong.

Write down what your baseline does well and where you already know it struggles. Speed isn't the point of a baseline. You want one you understand and can repeat, and the report template has the fields to start from.

A handful of real tasks

Pick five to ten prompts from your real work, each with a clear pass check. Put in your longest routine input and one task that has tripped a model up before. Store the input files beside the prompts so every run gets identical material.

Add without replacing

Adding tasks is fine as long as a fixed core stays put. That core is what lets you compare one version with the next, and it keeps you from setting a new prompt set against an old one without noticing.

Which numbers you need depends on the work. In chat you feel the wait before the first word, and the total time, far more than server throughput. For a batch job, record the finished workload and how many requests ran at once. Whatever you choose, keep the definitions the same every time. The Ollama speed guide shows how to read clean timings from one runtime.

One row per run

Keep it compact, one row per run with a link to the saved output. Failures and the odd slow run stay in. If you publish a median, say which runs went into it.

Know what your tool leaves out

Each tool times a slightly different slice of the work, and llama-bench is upfront about its own:

"The measurements with llama-bench do not include the times for tokenization and for sampling." (llama.cpp: llama-bench)

A llama-bench figure and a stopwatch on a chat app won't agree, and both can be right, so note which one you used. When you've missed a measurement, leave the cell empty. A number estimated after the fact makes everything around it harder to trust.

Judge on work done

Compare tasks passed and total time. Your single best run tells you very little. If something gets worse only on long inputs or on one kind of task, look into it before you give up the known-good setup.

One variable when you're hunting

Finding a cause means changing one thing and rerunning the same tests. Swap the whole stack at once and what you have is a comparison of two setups, which is enough to choose between them and won't tell you which change made the difference.

Keep the old configuration until the new one passes your checks, and note the date you switched and why. Do that after each meaningful update and the history builds itself, with no daily benchmarking. The quantization test uses the same method to choose between two files.

A regression you caught with both builds logged is worth passing on, since anyone on the same engine may run into it. When you share a result, give the settings and how you measured. The benchmark guide lists what readers check first.