Choosing a model
How to choose a local LLM for the hardware you already own
By llamaperf · · 6 min read
Updated
Quick answer
Choose a model that fits the machine you have and gets your tasks right at the context you need. Check that the wait is acceptable for daily use. Shortlist two setups in the calculator and run the same prompts on both before you download anything bigger.
Start with the job you need done
Write down what you want the model to do before you look at any leaderboard. "Help with code" is too vague to test, while "fix this failing function without changing its signature" is something you can check in a minute.
Build a small prompt set
Collect five or six prompts from your real week. Mix the easy questions you'd ask any day with one you've struggled with, and include one long input, because long inputs are where memory runs out first. Decide what a pass looks like for each prompt before you run anything.
For document work, use a file where you already know which details matter, so a missed one jumps out in seconds. Someone else's prompt set can't tell you which model suits your work.
Score the result apart from the writing
Keep two columns in your notes. One records whether the model did the task, the other whether you liked reading the answer, and only the first one decides anything. A polished explanation that ignores the output format you asked for still goes down as a miss.
Size the setup for the context you'll use
Open the VRAM calculator, pick your hardware and set the context to what you'll really send. Short chat turns need far less than whole reports pasted in. If you plan for the biggest window on offer, you'll rule out models that would've served you fine.
Check your runtime's default
Your runtime picks a context size as well, and it may not be the one you expect. Ollama's context length page says it defaults to 4k tokens on machines with under 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB and above. The same page is blunt about the cost:
"Setting a larger context length will increase the amount of memory required to run a model." (Ollama, Context length)
When you size a setup, read the calculator's whole memory breakdown, which includes the space inference needs on top of the weights.
Leave room for the rest of your machine
The calculator's figure is an estimate. Other apps and the exact model file both move the real number, so keep some headroom for the computer itself and check what your inference tool reports once the model has loaded.
Macs need more slack than the spec sheet suggests. macOS keeps part of the unified memory for itself, so a 32 GB Mac doesn't hand the model 32 GB. The Mac pages group community reports by chip and memory size, and the calculator shows how much of that memory it assumes the GPU can use.
Shortlist exact builds
You want two candidates: a small, quick one and a bigger one that still fits. For each, save the exact model name, the quantization, the runtime version and the context setting. A family name on its own doesn't tell anyone what you ran.
How much the quantization changes the size
The llama.cpp quantize README lists Llama 3.1 8B at 32.1 GB in its original form and 4.9 GB at Q4_K_M. For the 70B it's 280.9 GB against 43.1 GB. Specify the exact build before checking whether the 70B will run. Our quantization guide explains what the labels stand for.
Read reports from machines like yours
Browse hardware reports for setups close to yours, and open the original post when there is one. A report with a different quantization or a second GPU still tells you something about the hardware, though it won't predict your speed.
If nobody has reported your combination, write "unknown" in the speed column and measure it yourself. Keep nearby results as leads, rather than filling in a measured speed for your setup.
Pick the one that got the work done
Run your prompts through both options and mark which ones passed. Then time the wait before text appears and how long the useful answer took to finish. Do every run at least twice so a single lucky fast run can't settle it.
Keep the results in one table
| Question | What to record |
|---|---|
| Did it solve the task? | Pass criteria and any manual repair |
| Was the wait acceptable? | Initial wait and full completion time |
| Did it hold up on longer input? | Context setting and input length |
| Can you reproduce it? | Model, quantization, engine and hardware |
For anything else worth noting down, see the benchmark reading guide.
When the bigger model earns its place
If the smaller model passes as often, keep it. Its answers come back sooner and it leaves more memory free. The bigger one is worth keeping when it wins tasks the small one keeps failing and the wait doesn't bother you.
Hang on to the prompt set after you've chosen. When your work changes or a new build comes out, rerun it and compare against the same table. If you end up on a model, quant and machine that nobody has posted yet, submit it with the exact build and context you used.
Frequently asked questions
Is there a way to estimate tokens per second from my VRAM?
Not really. VRAM tells you what fits. Speed comes mostly from how fast the card can read the model's weights out of memory, so two cards with the same VRAM can be far apart. The llamaperf calculator estimates speed from memory bandwidth, but if someone has measured the same card and build, trust their report over any estimate.
What's the minimum tokens per second that's still usable?
No single number works for everyone. Reading along in a chat, you'll put up with a much lower speed than an agent can, since it has to read and act on every answer before its next step. Time a few of your own prompts and decide whether you'd sit through that wait every day.
How much memory does a model need?
Take the file size of the build you'd download and add the context cache plus some runtime overhead. For a sense of scale, the llama.cpp quantize README gives Llama 3.1 8B as 4.9 GB in Q4_K_M and the 70B as 43.1 GB.
Should I run a bigger model at a lower quant or a smaller model at a higher quant?
Check in the calculator which of the two fit your memory, then run whichever fit on your own prompts. Choose using the pass rate on your tasks.
Does the context length setting matter if I only ask unrelated questions?
Yes. Within one chat, every new message resends the earlier turns, so they count toward the context, while a fresh chat starts empty. Either way the setting caps how much memory the runtime may use, so size it for the longest conversation you really have.