How to run an LLM locally
Start with your computer's memory, then pick the program that runs models, then pick a model that fits it. This guide takes them in that order.
1. Check your memory
Memory is the limit that matters most. The whole model has to sit in memory while it runs. On a PC with a dedicated graphics card, that's the card's own memory (VRAM). On a Mac, it's the unified memory the chip shares with everything else, and macOS lets the GPU use only part of it by default.
A rough guide: at Q4_K_M with a 8K context, a dense model of up to about 6B parameters fits in 8GB with room to spare, 16B in 16GB and 28B in 24GB. A bigger model needs more memory, and so does a longer context. Our VRAM requirements guide has the full tables, and the LLM VRAM calculator checks a specific card or Mac against a specific model.
2. Pick a runtime
A runtime is the program that loads a model and talks to it. Two easy places to start are LM Studio, which gives you a window with a model browser and a chat box, and Ollama, which you drive from a terminal. Both download models for you. Our runtime pages show the speeds people report on each.
3. Choose a model that fits
Start with the biggest model that fits your memory with some room to spare, at a 4-bit quant. If you're not sure what a quant is, the quantization guide explains the names. To see what other people run on the same memory size, use the local LLM picker. For code, see the best local LLM for coding.
4. Know what speed to expect
Speed is measured in tokens per second, and it depends mostly on how fast your hardware moves data from memory. The benchmarks leaderboard groups reports by class of hardware. For how to read one fairly, see how to compare local LLM benchmarks.
5. Share your result
Once a model runs, the number you get helps the next person pick hardware. Benchmark your own machine and it joins the reports on this site.
Frequently asked
Can I run an LLM locally without a GPU?
Yes. The model runs on your CPU and system memory instead. It works, and it generates text more slowly. The leaderboard has a CPU-only tier that shows what people get.
How much memory do I need to run an LLM locally?
Memory decides which models load. By the calculator's estimate at Q4_K_M with a 8K context, a dense model of up to about 6B parameters fits in 8GB with room to spare, 16B in 16GB and 28B in 24GB.
Is a local LLM as good as ChatGPT or Claude?
Hosted models are far larger than anything that fits on one consumer card, so ask whether a smaller model handles your task. A local model has no per-request fee, runs offline and keeps your files on your machine. Try one on a real task of yours and judge the answers yourself.
Which runtime should I start with?
If you want a window to click through, try LM Studio. If you prefer a terminal, try Ollama. Each keeps its own downloaded models, so switching later means downloading a model again. The runtime pages show the speeds people report on each.