Choosing a model
MoE models: total vs active parameters
By llamaperf · · 6 min read
Quick answer
Plan memory by the total parameter count, and expect generation speed closer to a model the size of the active count. That suits machines with lots of memory and modest bandwidth, and it's why keeping the experts in system RAM works better than you'd expect.
Two parameter counts
Mixture-of-experts models carry two sizes in their names, like 30B-A3B: 30 billion parameters in total, 3 billion active for each token. One of those numbers decides your memory and the other decides your speed.
Total parameters set the memory
Each MoE layer holds many expert blocks, and a small router picks a few of them for every token. Any expert could be the next one picked, so they all have to be loaded. The Hugging Face MoE explainer states the memory requirement:
However, all parameters need to be loaded in RAM, so memory requirements are high. From Mixture of Experts Explained on Hugging Face
Active parameters set the work per token
For any one token, only the chosen experts do work, along with the parts every token shares, such as attention. Mistral put it this way for Mixtral:
Concretely, Mixtral has 46.7B total parameters but only uses 12.9B parameters per token. From Mistral's Mixtral announcement
In memory you pay for a 47B model, and in work per token for roughly a 13B one. The Hugging Face post explains why the total is 47B and not eight times 7B: only the feed-forward layers are split into experts, and everything else is shared.
Working out the memory
Use the total count
Plan weight memory from the total size, just as you would for a dense model. Take a 46.7B model at Q4_K_M, which averages about 4.9 bits per weight (our post on GGUF quant names explains why). 46.7 billion times 4.9 bits, divided by eight, is about 28.6 GB. That's the download and the memory for the weights alone, whatever the active count says.
Add the context
The KV cache belongs to the attention layers, which are shared, so an MoE model's cache is sized like any other model with the same attention shape. Our post on context length and the KV cache walks through that arithmetic. The VRAM calculator reads the total and active counts separately for sparse models, so put the model in there and check the fit at the context you really use.
Why speed tracks the active count
On a single-user setup, token generation is mostly limited by how fast your hardware can read the weights it needs. An MoE model reads only the active experts, which is far fewer bytes per token than a dense model of the same total size.
Fewer bytes read per token
Use the Mixtral figures as an example. At about 4.9 bits per weight, the 12.9B active parameters come to roughly 7.9 GB read per token, against about 28.6 GB if all 46.7B were read. On the same memory bandwidth that's more than three times fewer bytes. A big MoE model can feel quick on a machine with lots of fairly slow memory for exactly this reason, whether that's a Mac with unified memory or a PC with plenty of system RAM.
Where the estimate breaks
A long prompt, or several users at once, sends different tokens to different experts, so each step reads far more of the model, and prompt processing on an MoE ends up costing closer to its total size than its active size. Router overhead and uneven expert use take a bit more off the ideal figure. Start from the estimate and then look at the GPU reports for what people measure.
Keeping the experts in system RAM
Each token touches only a few experts, so you can keep the expert weights in system RAM and the shared layers on the GPU. The GPU handles the parts every token needs, and the CPU reads only the experts each token picks.
llama.cpp's --cpu-moe option
The llama.cpp server documentation lists --cpu-moe to "keep all Mixture of Experts (MoE) weights in the CPU", and --n-cpu-moe N to do the same for only the first N layers. The second lets you fill your VRAM with as many experts as fit and leave the rest in RAM. Plain layer offload is blunter, because it moves whole layers with their shared parts. Our post on partial offload covers that approach.
Test it on your machine
How well this works comes down to your system RAM speed and how many experts each token uses. Start with --cpu-moe and measure generation speed and time to first token on a real prompt. Then lower the number of layers whose experts stay on the CPU with --n-cpu-moe until VRAM is nearly full, and compare the result with a dense model that fits entirely on your GPU. A Mac has no split to tune, since its memory is unified, and the Mac reports show how MoE models run there.
Frequently asked questions
Do I need to fit the whole MoE model in VRAM?
All the experts have to be in memory, but that memory can be split. llama.cpp's --cpu-moe and --n-cpu-moe options keep the shared layers on the GPU and the expert weights in system RAM. It runs more slowly than keeping everything on the GPU, but only the selected experts need reading from system RAM for each token.
How much VRAM does an MoE model take compared to a dense model?
The same as a dense model with the same total parameter count at the same quantization. A 46.7B MoE at Q4_K_M needs about 28.6 GB for its weights, like any other 46.7B model. The active count changes the speed and nothing else.
Why is MoE generation faster than a dense model of the same size?
The router sends each token to a few experts, so the hardware reads far fewer weights per token. Mixtral uses 12.9B of its 46.7B parameters for each token, roughly a quarter of the bytes a dense 46.7B model would read.
How fast does an MoE run if only the active parameters fit in VRAM?
That depends on your system RAM bandwidth, since the experts get read from RAM for every token. Measure it on your own machine with the experts on the CPU, then compare against a dense model that fits entirely on the GPU.
Is a 30B MoE smarter than a 32B dense model?
Not automatically. That depends on the particular model, and the MoE will usually be the faster of the two. Run both on the tasks you care about and weigh the answers along with the speed.
Does prompt processing get the same speedup on an MoE model?
Less of it. In a long prompt different tokens go to different experts, so most of the model gets read during prompt processing, and its cost ends up nearer the total parameter count.