Hardware planning
Choosing a local LLM with 12 GB of VRAM
By llamaperf · · 6 min read
Quick answer
Everything on a 12 GB card draws on the same memory: the weights, the context, the runtime's overhead and anything else using the GPU, your desktop included. Pick a model that fits comfortably in the calculator and give it the longest prompt you'd send in a normal week. Before you try anything tighter, confirm the model is running entirely on the GPU.
What fits in 12 GB
Start from the file size
The llama.cpp quantize README lists real sizes for Llama 3.1 8B: 14.96 GiB at F16, 7.95 GiB at Q8_0 and 4.58 GiB at Q4_K_M. The F16 build won't load on a 12 GB card at all. Q8_0 fits with roughly 4 GiB to spare, and Q4_K_M leaves enough for a long context.
VRAM only narrows the list. Which model you end up with also depends on your task, your runtime and the context you plan to use, and a file smaller than your card can still fail to fit. Put a 10 GB file on a 12 GB card and you have 2 GB left for the context, with your desktop and browser drawing on the same GPU. If the quant labels are new to you, our quantization guide explains them.
Use the calculator for the rest
Memory bandwidth varies between 12 GB cards, so two of them can run the same model at very different speeds. Pick your exact card in the calculator if it's listed. It splits memory into weights, context and overhead, and the VRAM requirements guide explains each line. If your card is missing, a similar one will do for rough planning, as long as you remember the numbers are borrowed.
Test a long input early
For the test, use a document, conversation or code file about as long as the longest thing you send in a normal week. "Hello" proves the model loads and tells you nothing about your real work.
Your runtime's default context may be small
Ollama starts small on a card like yours. Its context length page says the default is 4k tokens when there's under 24 GiB of VRAM, so a 12 GB card gets 4k unless you raise it, and most of a pasted 30-page document won't make it in.
Note the context setting and how long your input was, and leave room for the answer. If the runtime cut the input short, write that down too. A run that only saw half the document hasn't passed the long-input test, however good the answer looks.
Shrink the cache if context is what you need
You can store the context cache at lower precision. According to the Ollama FAQ, setting OLLAMA_KV_CACHE_TYPE to q8_0 takes about half the memory of the default f16, and q4_0 about a quarter. When the cache is what fills your card, q8_0 roughly doubles the context you can fit in the memory that's left. Check your answers after the switch, because the same FAQ warns that q4_0 loses more precision at larger context sizes.
Then run the test twice, once with other GPU apps closed and once with your usual browser and editor open. The second run is the one that looks like an ordinary working day.
Check where the model is running
Most runtimes don't fail when a model doesn't quite fit. They move part of it to system memory without telling you, and your speed drops.
Read the Processor column
With Ollama, ollama ps shows where the loaded model lives, and the Ollama FAQ explains how to read the split:
"
48%/52% CPU/GPUmeans the model was loaded partially onto both the GPU and into system memory" (Ollama FAQ)
Save that output with your test results. A model at "100% GPU" and the same model split with the CPU are two different setups, even though they share a name.
What to do when it spills
Partial offload can still be worth running, particularly when the bigger model is much better at your task. Compare it against measured times for offloaded runs, since an all-GPU estimate doesn't describe it.
To get back onto the GPU, change one thing per attempt: a smaller quantization, or a shorter context that still covers the job. Rerun the same task each time and keep the earlier results, so you can see which trade-off got you to something usable. Benchmarking Ollama's generation speed shows how to time every attempt the same way.
Write down the trade-off you picked
When you settle on a setup, write down why. "Biggest model I could load" is the usual answer, and it leaves out whether the model passed your tasks and how long you waited for it.
A one-page decision sheet
| Field | What to write |
|---|---|
| Model build | Exact name and quantization |
| Context | Setting and the longest input you tested |
| Placement | 100% GPU, or the CPU/GPU split from ollama ps |
| Task results | Which prompts passed and what you fixed by hand |
| Timing | Wait before the first word and time to a finished answer |
Where to find more candidates
GPU reports show what people run on other 12 GB cards. Lean on the ones that state enough of their setup for you to compare. If you haven't picked your test prompts yet, how to choose a local LLM for your hardware covers building a set.
Your card may have no close match in the reports yet. If so, the sheet above, filled in, tells the next 12 GB owner more than any ranking without settings, so post it.
Frequently asked questions
What are the best models for a 12 GB VRAM card?
Whichever one passes your own tasks fastest, and only your tasks can tell you that. For sizing, dense models up to about 14B parameters fit at 4-bit with some context to spare, and 7B to 8B models fit at 8-bit. Look your card up in the calculator and the reports before you download.
Can I run a 20B or 30B model on 12 GB of VRAM?
Often, if you accept part of it living in system memory. A dense 30B at 4-bit is bigger than 12 GB, so the overflow runs from system RAM and generation slows down. Sparse mixture of experts models suffer less, since only a few experts run for each token.
How do I check if my model is fully on the GPU?
Run ollama ps and look at the Processor column. 100% GPU means the whole model is on the card. Something like 48%/52% CPU/GPU means part of it is sitting in system memory.
Should I use Q8_0 or Q4_K_M on a 12 GB card?
For an 8B model you can load either, and the llama.cpp README lists them at 7.95 GiB and 4.58 GiB. Q8_0 leaves less room for context. Try both on your own prompts, and if you can't tell the answers apart, keep Q4_K_M.
How much context can I fit on a 12 GB card?
There's no single figure, because it depends on the model's size and how its attention stores the cache. The calculator gives the context cache its own line. Ollama defaults to 4k on cards under 24 GiB, and switching the cache to q8_0 roughly halves what the context costs in memory.