Troubleshooting
When a model doesn't fit in VRAM: partial offload
By llamaperf · · 5 min read
Quick answer
The runtime keeps as many layers as fit on the GPU and runs the rest on the CPU from system RAM. Generation slows because every token still has to pass through the CPU layers. Check the split with ollama ps or in the llama.cpp startup log, then try a smaller quant that fits completely and compare the two.
What happens when it spills
You download a model that's a little too big for your card, and it loads anyway. That's partial offload, which llama.cpp lists as a core feature:
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity From the llama.cpp README
Layers split between GPU and CPU
A model is a stack of layers, and each token passes through all of them in order. With partial offload the first chunk of layers sits in VRAM and runs on the GPU, while the rest sits in system RAM and runs on the CPU. Ollama and LM Studio are built on llama.cpp and make this split for you whenever the model doesn't fit.
Why a small spill hurts so much
To generate a token, the hardware reads every weight once. System RAM is far slower to read than GPU memory, so a CPU layer takes much longer per byte than a GPU layer, and since the two parts run one after the other, their times add up.
As an arithmetic example, suppose the GPU part of a token takes 10 ms and the CPU part another 10 ms. You've dropped from 100 tokens per second to 50 with most of the model still on the GPU, and that's how people end up halving their speed by moving just a few layers to RAM.
Check where the layers went
Check the split before changing settings. Unexpectedly slow generation can come from a model running partly on the CPU.
In Ollama
Run ollama ps while the model is loaded and read the Processor column. Ollama's FAQ describes the mixed case like this:
48%/52% CPU/GPU means the model was loaded partially onto both the GPU and into system memory From the Ollama FAQ
You want to see 100% GPU. Anything else means part of the model is in system RAM, and that line belongs next to any speed you write down.
In llama.cpp
The startup log prints a line such as offloaded 30/33 layers to GPU. A 0/33 there means nothing reached the GPU, usually because the build has no GPU backend or the layer count is zero. Make sure the log names your GPU before you blame the model.
Setting the layer count
The -ngl option
Run llama.cpp directly and the number of GPU layers is yours to choose. The server documentation describes -ngl (also --n-gpu-layers) as the "max. number of layers to store in VRAM". Give it a number, or all to try putting everything on the GPU. If loading fails with an out-of-memory error, lower it a few layers at a time until the model loads, and note the highest value that worked.
Leave room for the context
Layers on the GPU compete with the KV cache for the same VRAM, so a layer count that loads fine at 4K can run out of memory at 32K. Our post on context length and the KV cache explains why. Set the context you need first and find the layer count after it, and test with a long prompt as well as a short one. Your desktop needs some VRAM too, since the display runs on the same card.
Offload or a smaller quant?
Offload keeps a bigger model within reach, and a smaller quant that fits completely is usually much faster. Which you want depends on whether the bigger model gets your work right often enough to be worth the wait.
Run the same prompt both ways
Pick one of your real tasks and run it twice, once on the bigger quant with partial offload and once on a smaller quant that shows 100% GPU. Our guide to GGUF quant names helps you choose the smaller file, and the VRAM calculator shows whether it fits with your context. Write down the time to a finished answer and whether the answer passed. If both pass, keep the faster one. If only the offloaded model gets it right, the wait may be worth it for that job.
Write down the split
When you compare numbers with someone else, the split matters as much as the card. Two people with the same GPU can report very different speeds because one of them had six layers on the CPU. The benchmark-reading guide lists what else to record. If you submit a report, paste in the layer split or the ollama ps line, so anyone reading the GPU pages can tell your run apart from a fully loaded one.
Frequently asked questions
How many layers can I offload to my GPU?
Whatever fits beside the context cache and what your desktop uses on the card. Start with all of them. If loading fails, drop a few layers at a time, and keep the highest count that loads with the context you really use.
Is the speed loss from moving layers to the CPU linear?
Moving a layer to the CPU adds more time than that layer took on the GPU because system RAM is slower to read. A handful on the CPU can take as long as every GPU layer put together, which is how speed halves with most of the model still on the GPU.
Why does llama.cpp say offloaded 0 layers to GPU?
It means nothing went to the GPU. The usual causes are a build without a GPU backend or a layer count of zero. Look for your GPU's name in the startup log, and if it's missing, install a build with CUDA, ROCm, Vulkan or Metal support for your hardware.
How can I tell if Ollama is running my model on the GPU?
Run ollama ps with the model loaded and look at the Processor column. 100% GPU means everything is in VRAM and 100% CPU means it's all in system memory. A split like 48%/52% CPU/GPU means the model is divided between the two.
Is it better to offload a bigger model or run a smaller quant fully on the GPU?
Try both on a real task. The smaller quant usually wins on speed by a wide margin, so the offloaded model has to earn its place by getting answers right that the smaller one misses, with a wait you don't mind.