llamaperf
← All articles

Hardware planning

Serving one local LLM to several users: slots and memory

By llamaperf · · 5 min read

Quick answer

One loaded model can answer several people at once through parallel slots. Every slot needs its own context memory, so four slots with an 8K context need room for 32K tokens of cache. Total output rises as more people use it, while each person's answers come out slower. Measure with as many people as you expect, all asking at the same moment.

How parallel slots work

The server keeps one copy of the model in memory and gives each request a slot of its own, with its own conversation state.

Ollama processes one request at a time by default

An environment variable controls this, and Ollama's FAQ gives the default:

The maximum number of parallel requests each model will process at the same time, default 1.

From the Ollama FAQ

So out of the box, a second person's request waits in line until the first answer finishes. Raise OLLAMA_NUM_PARALLEL and Ollama works on that many requests together.

llama.cpp calls them slots

In llama-server the setting is -np (or --parallel), and the server README describes it as the number of server slots. Each slot is one conversation the server can work on at the same time. Set it near the number of people who'll really be typing at once, which is usually fewer than the number who have access.

Every slot needs its own context

The weights are shared by everyone. The cache of tokens each slot has seen belongs to that slot alone, and it takes memory.

Multiply the context by the slot count

Ollama's FAQ is direct about it: required RAM scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. If one person at an 8K context fits comfortably, four slots at 8K need cache room for 32K tokens on top of the weights, and that's often what pushes a setup past the card's memory. Our post on context length and KV cache memory explains why the cache grows the way it does, and the calculator shows its size for a given model and context.

Ways to fit more people

If your users send short questions, give each slot a shorter context. Quantizing the KV cache saves memory as well, and the Ollama FAQ puts a q8_0 cache at about half the memory of the default. A smaller model file frees room too. Change one thing at a time and check the answers still come back correct after each.

Per-user speed and total throughput

The total rises while each reply slows

With four people asking at once, the server gets more done per second overall, because one read of the weights serves several requests. Each individual answer still comes out more slowly than it would for one person alone. Measure per-user latency as well as the combined figure to see how long each person waits.

Test with real concurrency

Send as many requests as you expect at the same time, from a small script or a few browser tabs, with prompts of realistic length. Record each person's wait before the first word and output speed. Keep the total throughput alongside those readings. Then run it again with a single user so you have a baseline. When you share the result, say how many requests ran together. Our performance guide explains why a multi-user total and a single-user speed belong in separate columns.

Plan for the busy moments

Plan for requests that arrive while every slot is busy.

What happens when the slots are full

Extra requests wait in a queue. Ollama's FAQ says it holds up to 512 by default before it starts refusing, and an overloaded server answers with a 503 error. A handful of people will rarely get near that. A slow model writing long answers can make the wait feel long well before the limit, though, and in that case fewer slots with shorter contexts sometimes feel faster than many slow ones.

When a serving engine makes sense

For a small group, Ollama or llama-server is enough. Serving a team through an API all day is a heavier job, and an engine built for it is worth a look. The vLLM README lists continuous batching of incoming requests among its features, which keeps the GPU busy as requests come and go. Before you buy hardware for this, look through GPU reports for multi-user results. If you run a concurrency test of your own, submit it with the number of simultaneous requests written in, so nobody reads your total as a single-user speed.

Frequently asked questions

Can a single loaded model handle parallel requests?

Yes. Ollama and llama-server both keep one copy of the model and serve several requests at once through parallel slots. Ollama handles one at a time until you raise OLLAMA_NUM_PARALLEL.

How many concurrent users can a local LLM handle?

That comes down to your memory and how long everyone's context is. Each extra slot needs its own cache, so work out the memory for one user and multiply by the number of users. Then test with that many people asking at once.

How do parallel requests share context size in llama.cpp?

llama-server can keep a single KV cache buffer that every slot draws from. According to its README, that unified buffer is on by default when the slot count is automatic, and the kv-unified-per-slot option caps how much any one slot can take.

Does OLLAMA_NUM_PARALLEL use more memory?

Yes, roughly in proportion. The Ollama FAQ says required RAM scales by OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH, so doubling the slots about doubles the memory the context needs.

Why is each user slower when several people use the model?

The GPU is splitting its work between the requests. Total output goes up, but each person's share of it goes down, so any one answer takes longer than it would if you had the model to yourself.

Should I use vLLM instead of Ollama for multiple users?

Not for a few people, where Ollama or llama-server does fine. A team hitting an API all day is the load vLLM and its continuous batching are built for. If you're on the fence, test both with your real traffic.