llamaperf
← All articles

Hardware planning

Running a local LLM on CPU only: what to expect

By llamaperf · · 5 min read

Quick answer

Yes, a CPU alone can run a local model, and small quantized models are pleasant to use that way. How fast the answer comes out depends on your memory bandwidth, since every new token reads the whole model from RAM. Start with a small quantized model, time the prompt and the answer separately, and leave the GPU decision until you've seen those numbers.

You can run a model with no GPU at all

llama.cpp was written to run on ordinary processors first, which is why plenty of people run local models on a laptop or an old desktop with no graphics card in it. Its README lists the instruction sets it uses on x86 chips:

AVX, AVX2, AVX512 and AMX support for x86 architectures

From the llama.cpp README

What you give up

You mainly give up speed. A CPU can do the maths, but it reads memory far more slowly than a graphics card reads its own VRAM, and reading memory is most of what generation does. Long prompts take longer too. Processing your input is heavy arithmetic, and a desktop CPU has much less of that to spare.

When CPU-only is the right call

Short questions work well, and so do jobs you're happy to leave running in the background while you get on with something else. It's also the cheapest way to find out whether you'll use local models at all, so try it on the machine you already have before you spend money on a card. If you later want to know where a GPU would change things, our performance guide splits a run into its parts and says which piece of hardware speeds up each one.

Memory bandwidth sets the ceiling

Text comes out one token at a time, and for every token the model reads essentially all of its weights from memory. Two things set how fast text can appear: the size of the model file and the speed of your RAM.

Smaller files come out faster

That's why quantization matters so much on a CPU. The llama.cpp quantize README lists Llama 3.1 8B at 32.1 GB in its original form and 4.9 GB at Q4_K_M, so each token means reading less than a sixth of the data. If the labels on the files mean nothing to you yet, our post on what Q4_K_M means decodes them.

Some rough maths with made-up numbers shows how this plays out. Say your RAM delivers 60 GB/s and the model file is 5 GB. You can read that file twelve times a second, so generation tops out around 12 tokens per second, and a real run lands somewhere below that. Look up your own memory speed and redo the sum, or let the calculator do it for the hardware it knows.

Why extra cores stop helping

Once the cores are all waiting on memory, more threads do nothing for generation. In a typical desktop, dual-channel memory limits generation speed. More memory channels or faster RAM can raise that limit. A newer processor on the same memory setup often helps less.

Test it on your own machine

Use a small model at a 4-bit quantization for the first run, so it finishes quickly, and only move up in size once you have a baseline.

Time the prompt and the answer separately

Reading your prompt leans on the CPU's arithmetic and writing the answer leans on memory, so one combined time hides which part is slow. Long prompts are where a CPU hurts most. Our post on measuring time to first token gets both numbers out of a single streaming request.

Test thread count as well. Start at the number of physical cores and work downward, keeping whatever gives the fastest generation on your machine. Hyperthreads rarely help a job that spends its time waiting on memory.

Write down what else was running

A browser full of tabs competes with the model for the same memory bandwidth. Note your RAM size and speed, how many memory channels you have and what else was open, because without those details nobody can line their result up against yours. CPU-only results are rare in most collections, so clean numbers from your machine are worth submitting.

Decide whether you need a GPU

When a card starts paying off

If you keep waiting a minute or more for the first word on long documents, a GPU will shorten that wait more than anything else you could buy. A GPU is also worth considering when your RAM cannot feed a larger model at a readable pace. And when a model almost fits on a card you already own, you can split its layers between GPU and CPU, which our post on models too big for VRAM walks through.

Sparse mixture-of-experts models deserve a look on a CPU because they read only part of their weights for each token. Our post on MoE memory and speed explains what you give up for that. Before you buy anything, browse GPU reports from people running the models you want, and compare runs of the same model at the same quantization.

Frequently asked questions

Can you run a local LLM without a GPU?

Yes. llama.cpp and the tools built on it run on ordinary x86 and ARM processors. A small quantized model answers at a readable pace on a modern desktop, and bigger ones work as well if you don't mind waiting.

How many tokens per second can I get on CPU only?

Mostly that comes down to your memory bandwidth and the size of the model file. Divide the first by the second for a rough ceiling, then measure, since real runs come in below it.

Does RAM speed matter more than CPU cores for local LLMs?

For writing the answer, yes. Generation waits on memory, so faster RAM and more channels beat extra cores there. Prompt processing is where more cores still pay off.

Is DDR5 worth it for CPU inference?

Compared with slower DDR4 on the same number of channels, yes, because faster memory raises the ceiling on generation speed. A board with more channels raises it further. If you can, measure before and after the upgrade.

What is the best small model to run without a GPU?

Whichever small model does your task well, run at a 4-bit quantization. What counts as well depends on what you ask it, so try two or three candidates on your own prompts.

How many threads should I use for llama.cpp on CPU?

Your physical core count is the place to start. Try a few lower values and keep the fastest. Going above the physical core count rarely helps, because the extra threads just sit waiting on memory.