Hardware planning
Why a longer context needs more memory
By llamaperf · · 6 min read
Quick answer
The model keeps a key and a value for every token in the context, in every layer, so the cache grows in step with the context you set. Make the window as long as your longest routine input and no longer. If memory is still tight, Ollama's 8-bit KV cache takes about half the space of the default 16-bit one.
What the KV cache holds
A language model writes one token at a time, and every new token looks back at all the tokens before it. Runtimes hold on to the intermediate results of that lookback so they don't recompute the whole history on every step.
A KV cache stores these calculations so they can be reused without recomputing them. From the Hugging Face Transformers cache documentation
One entry per token, per layer
Each attention layer stores two vectors for every token in the context, a key and a value, which is where the name comes from. In a model with 32 layers that's 32 pairs for every token you've sent and every token it has written back. The weights stay the same size whatever you do. The cache keeps growing for as long as the conversation does.
Why it grows in a straight line
Take a model with 32 layers, 8 key-value heads and a head size of 128, storing each value at 16 bits (2 bytes):
2 (key and value) x 32 layers x 8 heads x 128 x 2 bytes = 131,072 bytes per token
131,072 bytes x 32,768 tokens = 4 GiB
Double the context and the 4 GiB doubles with it. Those are example numbers, and your own model's shape is in its config file. The Hugging Face page warns that the cache "can occupy a significant portion of memory and become a bottleneck for long-context generation."
The window you set and the text you send
Runtimes reserve the window up front
The context window you configure and the length of what you send are two different numbers, and memory follows the first one. llama.cpp sizes its cache for the context you pass with -c when the server starts, so a 32K setting costs 32K worth of memory even if you only ever ask short questions. Ollama's FAQ gives 4096 tokens as its default window, which you raise with OLLAMA_CONTEXT_LENGTH or the num_ctx parameter. That small default is why Ollama can quietly cut off a long document until you change it.
Parallel requests multiply cache memory. The same FAQ says required RAM scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH, which means four parallel slots at 16K need as much cache as one 64K window.
Some models stop growing
Some layers don't keep the whole history at all. With sliding-window attention those layers hold only the last few thousand tokens, and the Hugging Face docs note that for them "the cache will stop growing when the layers using these types of attention have reached their maximum size". Two models of the same size can therefore need very different amounts of memory at 32K.
Making the cache smaller
When the weights fit and the context doesn't, there are two settings to try before you settle for a smaller model.
Quantize the KV cache
Ollama stores the cache at lower precision if you set the OLLAMA_KV_CACHE_TYPE environment variable. Its FAQ spells out the trade: q8_0 uses about half the memory of the default f16 with a very small loss in precision, and q4_0 uses about a quarter, with a loss that may show more at long contexts. It's a global setting, so every model you load picks it up. llama.cpp does the same thing with -ctk and -ctv, the cache types for keys and values.
Start with q8_0 and rerun whichever task leans hardest on details from early in a long input. A lower-precision cache tends to slip there first.
Turn on Flash Attention
Ollama won't quantize the cache unless Flash Attention is on, and the FAQ says Flash Attention by itself "can significantly reduce memory usage as the context size grows". Recent versions turn it on automatically when your hardware supports it. OLLAMA_FLASH_ATTENTION=1 forces it.
Size the context to your work
Measure your longest input
To avoid reserving unused memory, take the longest thing you paste in a normal week, a contract or a long chat log, and count its tokens with your runtime or tokenizer, then add room for the answer and a margin on top. Quick questions are fine at 8K. Document work might need 32K. Setting 128K just because the model supports it spends memory that could have gone to a better quant.
Check the fit before you commit
Put your card, the model and that context length into the VRAM calculator. It lists weights and cache separately so you can see which allocation fills the card. If it's the weights, our post on reading GGUF quant names will help you find a smaller file. On a Mac, the model and cache share memory with every open app. Look at the Mac reports from people with the same memory size as yours. The VRAM requirements guide covers the rest of the budget.
Frequently asked questions
How do I find out how much VRAM the context needs?
Get the bytes per token first: two, times the layer count, times the key-value head count, times the head size, times the bytes per value. Multiply that by your context length and you have the cache. If you'd rather skip the arithmetic, the llamaperf calculator shows the cache next to the weights.
How much can I quantize my KV cache before quality drops?
Ollama's documentation puts q8_0 at a very small loss for half the memory, and q4_0 at a small to medium loss that shows more at long contexts. Start at q8_0, and test it on the task that depends on details from early in a long input before you go any lower.
Does the context length setting matter if I only ask short questions?
For memory, yes. llama.cpp reserves the cache for the whole window at startup, so a large setting costs memory however short your prompts are. The answers to short questions come out the same either way.
What's a good context length for a personal assistant?
Whatever your longest routine input is, plus room for the reply. Short chats rarely need more than 8K. If you paste in documents or long code files, measure the biggest one and set the window a little above it.
Why does my model use more memory than the file size?
The file is only the weights. At runtime the model also needs the KV cache for your context, plus compute buffers, and at long contexts the cache by itself can run to several gigabytes.