llamaperf
← All articles

Measurement

How to measure tokens per watt on your local LLM

By llamaperf · · 5 min read

Quick answer

Log your GPU's power once a second with nvidia-smi while you time a run. Average watts times run time gives you joules, and joules divided by the tokens produced gives energy per token. Compare setups on that figure with the same prompt and output length, and keep in mind that the GPU reading leaves out the rest of the computer.

Measure energy per answer

A card's wattage by itself tells you little. A fast card that draws more power can still use less energy per answer, because it's done sooner.

Watts are a rate, your bill counts energy

Power is how fast energy gets used, and energy is power times time. For a local model the useful figure is energy per token, in joules, or its inverse, tokens per joule, which is what people usually mean when they say tokens per watt. This lets you compare a short, high-power run with a longer, low-power run.

Idle draw counts too

A card that sits loaded all day waiting for the next question still uses power in between. If your model runs around the clock, note the idle reading as well as the one under load. On a machine that answers a few questions an hour, the idle figure can matter more.

Log power with nvidia-smi

Use nvidia-smi to log repeated power readings from an NVIDIA card.

The command

nvidia-smi --query-gpu=timestamp,power.draw.average --format=csv -l 1 > power.csv

That reads the following field, as defined in the nvidia-smi documentation:

The average power draw for the entire board for the last second, in watts.

From the nvidia-smi documentation

According to the same page, the field is supported on Ampere (except GA100) and newer cards. On an older card, query power.draw instead, which gives the last measured reading. The -l 1 sets a one-second interval. Leave it out and you get one reading and nothing more, and the docs say -l with no number defaults to every 5 seconds, which is too coarse for short runs.

Line the log up with the run

Start the log and let it record a few seconds of idle baseline, run your prompt, then give it a few more seconds afterwards. Keep the timestamps from the log and from the run, so you can cut out exactly the seconds the model was working. The other half of the calculation is a timed run with a token count, and our post on measuring time to first token shows how to get both from a streaming request.

Turn the log into joules per token

The arithmetic

Average the power readings over the seconds the model was generating, multiply by the generation time and divide by the tokens generated. The Ollama API reference hands you the other two numbers directly: eval_count is the output tokens and eval_duration is the time spent generating them, in nanoseconds.

An example with made-up numbers: a run averages 300 W for 20 seconds and produces 600 tokens. That's 6,000 joules for 600 tokens, or 10 joules per token, which comes out at 0.1 tokens per joule if you flip it.

What the GPU reading leaves out

nvidia-smi measures the card and nothing else. Your CPU, memory, fans and power supply losses all sit outside that number, and with partial CPU offload the CPU may be doing a lot of the work. A plug-in wall meter covers the whole machine, at a coarser interval. Whichever you used, say so when you share a result.

Compare setups fairly

Hold the workload fixed

Energy figures only compare when the work behind them is the same. Give every setup you test the same prompt and the same output length, and run each one a few times. Prompt processing and generation draw different power, so a long prompt with a short answer gives a different energy per token from the reverse. Write both lengths down.

Power limits change the result

The nvidia-smi docs note that you can adjust the power limit with -pl. Speed and power don't fall in step, so a lower limit can improve energy per token, and measuring your own card is the only way to find out by how much. If you ran with a limit, it belongs in your report next to the quantization. The report template has a space for it, and our benchmark-reading guide covers the other fields. A capped card can look like a slower one when people browse GPU reports, so put the limit where they'll see it when you submit yours.

Frequently asked questions

How do I measure tokens per watt for a local LLM?

Log the GPU's power once a second while you time a run. The average watts times the generation time is your joules. Divide the tokens by the joules for tokens per joule, or the joules by the tokens for energy per token.

Does nvidia-smi show the power draw of the whole computer?

No, just the card. CPU, memory, fans and power supply losses aren't in it. For the whole machine you need a plug-in wall meter.

Does lowering the GPU power limit slow down inference?

It can, but speed and power don't fall in step. Measure whether the energy saving is worth the change in speed. Try your own card at two or three limits, and mention the limit when you share results.

How much power does a GPU draw when a model is idle?

That varies with the card and the driver. Log it with nvidia-smi while the model sits loaded with nothing running, and include the figure if your machine spends most of the day idle between questions.

Is a Mac more power efficient for local LLMs?

Don't assume it either way. On a Mac, the powermetrics man page says its power values are estimated and shouldn't be used to compare different devices. Read a Mac's figures as a rough guide, and use a wall meter when you want to compare machines.

Why do my tokens per watt numbers change between runs?

Prompt length, output length, temperature and whatever else the machine is doing all move the number. Hold the prompt and output length fixed, repeat each run a few times and report the median.