Hardware planning
What to check before buying a second GPU for local LLMs
By llamaperf · · 5 min read
Quick answer
Start by naming what's stopping you: either the model won't fit, or it fits and answers too slowly. Check that your runtime supports the split you have in mind, and look for reports of that exact setup. A second card mostly buys memory. What it does for single-user speed depends on the engine, and the gain is often small.
Name the problem you're paying to fix
Capacity and speed are different purchases
What can't your current setup do? Maybe the model you want won't load. Maybe it loads and every answer drags. Those call for different purchases, and a second GPU is a good fix for the first one. If you serve other people and requests queue up, you have a third problem with fixes of its own.
vLLM's documentation opens its multi-GPU advice with the obvious case:
"Single GPU (no distributed inference): if the model fits on a single GPU, distributed inference is probably unnecessary. Run inference on that GPU." (vLLM: parallelism and scaling)
Save a baseline first
Record your current setup before you buy anything: the exact model, quantization, context and request pattern you care about, with task results and total time. Skip it and you'll have no way to tell whether the new card fixed anything. While you're there, try a smaller model or quantization that fits the card you already have. It costs nothing, and sometimes it's good enough. The 12 GB guide walks through testing a tight fit.
Check the whole setup will work
Your runtime has to support the split
Read the current docs for your engine with the exact pair of cards you plan to run in mind. llama.cpp, for example, splits by layer by default, also offers row and tensor modes, and takes --tensor-split to weight how much each card gets (llama.cpp server README). In layer mode each card runs its own layers in turn, so with one user the cards spend most of their time waiting on each other. That's why a second card adds room far more reliably than it adds speed. The vLLM guide quoted above covers when to split a model with tensor parallelism and when to add pipeline parallelism on top.
Some engines handle mixed cards and some don't. Where they do, the slower card tends to set the pace, so find out which kind yours is before you order.
The physical build
Check slots, spacing, power connectors, PSU headroom and cooling against the manufacturers' specs for the parts you'll buy. A card that fits on paper and then runs hot halfway through a long generation still needs work. Write down the whole working setup you're buying toward, power and cooling included.
Find reports of the same setup
Match the split
Browse the GPU reports and click through to the original posts. Plenty of reports are multi-GPU, and most of those use exactly two cards. Get as close as you can on model, quantization, engine, card count and input size, and check whether the post timed one request or several at once.
Keep the unknowns visible
Two different cards and two matching ones are separate setups, and a report that also spills layers to the CPU is another again. Keep them apart in your notes. The comparison pages put two cards side by side.
Some reports only show that a model loaded. That's evidence about capacity and tells you nothing about how long you'll wait, so write "speed unknown" beside it and keep looking.
Compare the options against your baseline
A short decision sheet
You need three rows: what you have now, a smaller model on what you have now, and the upgrade. Next to each goes the cost and the evidence behind any speed you expect. The calculator shows what fits in the combined memory.
If nobody has measured that hardware, leave the guess out. Link a matching report or write "not measured", which at least tells you what to check next.
Test before you commit if you can
If you can borrow a card or rent a similar machine by the hour, run your baseline prompts on it with the same pass checks, and see whether it fixes the problem you wrote down at the start. The performance guide defines the metrics, so your before and after numbers measure the same thing.
Frequently asked questions
Is it worth running two 12GB GPUs?
It can be, if you need about 24 GB to fit a model and no single larger card comes in at a similar price. The pair will load bigger models. Single-user generation won't get much faster, because the cards mostly take turns.
Does a second GPU make token generation faster?
For one user, usually not by much. llama.cpp's default layer split has each card process its layers in turn. On matching cards a tensor parallel engine can speed things up, and the bigger gains come when you serve several requests at once.
Can I use two different GPUs in one PC for local LLMs?
Often, yes, depending on the engine. llama.cpp will split a model across mismatched cards and lets you set each card's share. Expect the slower card to set the pace, and look for reports of your exact pair before buying.
Do I need NVLink for two GPUs?
No. llama.cpp and vLLM both run over PCIe. A faster link matters most for tensor parallel splits, since those exchange data between the cards on every layer.
Is a second GPU better than offloading to system RAM?
For speed, yes, as long as the model fits across both cards. Offloaded layers run from system memory, and that's much slower than VRAM. Compare reports of both setups on the same model before you decide.
How do I tell what my setup is limited by?
See whether the model loads fully on the GPU. Then time one request on its own, and time several at once. If the model won't fit, that's a memory problem. Slow single answers and long queues are two other problems, and they have different fixes.