Troubleshooting
Why long prompts can make a local LLM feel slow
By llamaperf · · 6 min read
Quick answer
Time the wait before the first word apart from the time spent writing the answer. Run a short and a long version of one real input with nothing else changed, then look at memory use and where the model is running. The context size you configure and the length of the prompt you send are separate numbers, and you'll want both.
Pin down what "slow" means
A long pause before anything appears, text that crawls once it starts, an answer that takes ages because it's twice the length it needs to be: people call all of these "slow", and each has a different cause. Decide which one you're chasing and write it down.
Keep the request fixed
On document work, keep the instruction identical from run to run. A two-line extraction in one run and a full explanation in the next are different jobs. Compare one request at several input lengths before you blame the hardware.
Pick one way to time the first word
A stopwatch, timestamps in a streaming client and the server's own metrics each start and stop at slightly different moments. Say which one you used for the wait before the first word, and use it every time, even if another method would give you an extra decimal place. Benchmarking Ollama's generation speed shows where Ollama reports each timing.
Your prompt and your context window are different numbers
Your context setting is the most the runtime will accept. Your prompt is however long the input really is, and in a chat app it includes the whole conversation so far. Record both, along with the output limit.
Why a longer context costs memory
The model stores every token it has read so it doesn't have to recompute it, and that store grows with the input. The Hugging Face documentation on KV cache strategies states the problem directly:
"The KV cache can occupy a significant portion of memory and become a bottleneck for long-context generation." (Hugging Face, Cache strategies)
From the runtime side, the Ollama context documentation says a larger context length increases the memory needed to run a model. Some engines reserve memory for the full window up front whether you fill it or not, so look up how yours behaves. The VRAM requirements guide shows how the cache adds up.
A big window doesn't prove the model read it
Put in at least one task where you can check that the model found a detail from the middle of the input. If it answered fast and missed the detail, the speed hasn't bought you anything.
Run a small length experiment
Pick three inputs from your real work at different lengths. Keep the instruction, model build, quantization and output limit the same for all three, and leave everything else on the machine alone while you test.
What to record
For each run, write down the wait before the first word, how long generation took, how many tokens came out and whether the task passed, plus memory readings if your runtime reports them. Run the whole set twice and note any warmup or cache reuse.
Both stages slow down as the input grows. In the example table in the llama-bench README, a 7B model processes a 512-token prompt at 7,340 tokens per second on an empty context and at 6,426 once 512 tokens are already in it, while generation goes from 120.6 to 116.7. That's a small drop at 512 tokens. It grows with depth, and tens of thousands of tokens in, you'll feel it.
What the results point to
| What you see | What to check next |
|---|---|
| Initial wait grows, output pace stays the same | Prompt processing figures |
| Output slows too, or the model moves off the GPU | Memory and offload settings |
| Timing looks fine, but facts go missing | Answer quality and truncation |
| An identical repeat is suddenly much faster | Prompt caching in the test |
Use the table to decide where to look next, and hang on to the raw numbers until you've confirmed the cause.
Cut the work while keeping the task intact
With the slow stage found, the aim is to give it less work while still getting the same answer back.
Trim the input
Cut the parts of the input that don't bear on the task, ask for a shorter output, or split the job into pieces you can check one at a time. Compare the answers along with the timings. A summary that arrives in half the time is no use if it dropped the one clause you needed.
Change one setting or prompt strategy at a time, and keep the previous version so you can go back to it.
When you need the whole input
Some jobs need every page. For those, use the calculator to see what the longer context costs in memory, then try it on your machine. If you're on a smaller card, choosing a local LLM with 12 GB of VRAM covers shrinking the cache.
A speed figure without the input length behind it is hard for anyone else to use. On the same setup, someone feeding in a 50-page contract waits far longer for the first word than someone asking a one-line question. The performance guide explains why, and the submission form has a field for it.
Frequently asked questions
What is the link between VRAM and context length?
Every token the model has read goes into a cache held in memory, usually VRAM. More context means a bigger cache, so the same model needs more memory at 32k tokens than at 4k.
Why does Ollama use more memory when I raise the context length?
Ollama sizes the context cache from that setting, and its own documentation says a larger context length increases the memory needed to run a model. Once the total stops fitting on the GPU, part of the model moves to system memory.
Is slow prompt processing normal when the model doesn't fit in VRAM?
Yes, it's common. Every prompt token has to go through the layers sitting in system memory, on the slow side of the split. A smaller quant or a shorter context that gets the whole model back on the GPU usually brings the speed back.
How do I speed up slow prompt eval?
Send less text and keep the model entirely on the GPU. If your engine caches prompts, keep the start of each prompt the same between requests so it can reuse that part. Measure after every change, or you won't know which one helped.
How much VRAM does the context alone need?
There's no general answer. It depends on the model's layer count and how its attention stores the cache, and two models of the same size can be far apart. The llamaperf calculator shows the context cache as a separate line for each model.
Does a long conversation slow down every reply?
A bit. The whole history goes back in with each message, so the input keeps growing. Engines with prompt caching skip the part they've already processed, but generation still slows slightly as the cache gets deeper.