Local model performance: what makes an LLM fast or slow

A local model generates one token at a time, and for every token the GPU has to read every active weight out of memory. So generation speed is, to a first approximation, memory bandwidth ÷ bytes of weights read per token. The hardware fixes the first number. Quantization shrinks the second. Everything else is a correction on top of that ratio.

The practical consequences: a card with more VRAM and similar bandwidth runs bigger models at the same speed, a card with more bandwidth runs the same model faster, and a model that does not fit in VRAM falls off a cliff, because the layers that spill into system RAM run at DDR speed.

Generation speed, in tokens per second, is how fast the reply streams out. It is the number people quote, and the one every report on this site records where the poster gave it.

Prompt processing speed, also in tokens per second, is how fast the model reads your input before the first token appears. It is compute bound rather than bandwidth bound, so the ranking of hardware can differ. It matters most for long documents, large codebases and agent loops that resend a big context every turn.

Neither number alone tells you how long an answer takes. A 40 t/s card with slow prompt processing can feel worse on a 30k-token prompt than a 25 t/s card that reads it quickly. Our guide on how to compare local LLM benchmarks covers what to check before putting two figures side by side.

Modern GPUs have far more arithmetic than a single conversation can use. Generating one token for one user is a long sequence of matrix-vector products, each of which reads a slice of the weights once and does very little maths per byte. The card spends its time waiting on memory, and the fastest it can go is bandwidth divided by the bytes it has to move.

That is why an RTX 3090 and an RTX 4090 land within a few tens of percent of each other on the same model despite the 4090's much larger compute budget, and why Apple Silicon can hold a 70B model in unified memory and still generate slowly: the pool is huge, the bandwidth is closer to a mid-range discrete card.

Our VRAM calculator estimates speed with exactly this roofline: the GPU's memory bandwidth divided by the gigabytes read per token, scaled by an efficiency factor calibrated against real runs. Where the community has measured the same model on the same card, the measured figure replaces the estimate and is labelled as such.

• Model size. Bytes per token scale with the number of active parameters. A mixture-of-experts model only reads its active experts, which is why a 30B MoE with 3B active can run several times faster than a dense 30B on the same card.
• Quantization. Fewer bits per weight means fewer bytes per token. Q4 reads roughly half of what Q8 reads, so it is roughly twice as fast, and it fits models that otherwise would not. See quantization for local LLMs.
• Context length. The KV cache grows with context and is read on every token. A long context costs memory, which can push a model out of VRAM, and costs speed even when it fits. The VRAM requirements guide shows how much headroom to leave.
• Engine and settings. llama.cpp, vLLM, MLX and ExLlama share the same bandwidth ceiling but differ in how close they get to it, and in features such as speculative decoding, KV-cache quantization and batching. Compare them on the engines page.

The one thing you cannot tune is the hardware's bandwidth. If the numbers below are not enough for the models you want, the fix is a different card, not a different flag.

Median generation speed reported to llamaperf for each GPU and Mac, across every model and quant people have run on it. Single-GPU reports only, and a card needs at least 3 of them to appear. The spread is wide because a 4B model and a 120B model on the same card are both in here, so treat the median as a feel for the card and open its page for like-for-like numbers.

Bandwidth is the column to read against the median. The fastest figure on a card is usually a small model or a batched serving run, not what one person sees in a chat.

• VRAM calculator estimates which models fit your card, at which quant, and how fast they should run.
• GPU pages list every report on a card, with model, quant, engine and context.
• Compare two GPUs side by side on the models they have both been reported running.
• Leaderboard ranks the models people run most within each hardware tier.

What does local model performance mean?

How fast an open-weight LLM runs on hardware you own, measured in tokens per second. Generation speed is how quickly the reply appears; prompt processing speed is how quickly the model reads your input. It is a different question from how smart the model is, which leaderboards such as MMLU or LMArena measure.

What is a good tokens per second for a local LLM?

For chat, anything above about 15 tokens per second feels immediate, because it is faster than most people read. Coding assistants and agents that write long outputs benefit from much more. Below about 5 tokens per second a reply visibly crawls, which is what happens when a model no longer fits in GPU memory and spills into system RAM.

Does a faster GPU always run local models faster?

No. Generation speed is limited by memory bandwidth, not compute, so a card with more TFLOPS but similar bandwidth barely helps. And if the model does not fit in VRAM, the layers that spill to system RAM run at DDR speed, which is far slower than any GPU. A slower card with enough memory beats a faster card that has to offload.

Why does a local model slow down with a long context?

Every generated token attends over the whole context, and the KV cache that stores it grows with context length. That adds bytes to read per token and memory pressure on the GPU. Prompt processing of a long input also takes time before the first token appears.

Does the inference engine change local model performance?

Yes, but less than hardware and quantization do. llama.cpp, vLLM, MLX and ExLlama read the same weights from the same memory, so they share the bandwidth ceiling. They differ in how close to that ceiling they get, in batching, and in which quant formats and GPUs they support.

Is local model performance the same as model quality?

No. Performance here means speed and fit on your hardware. Quality is how good the answers are. Quantization trades a little of the second for a lot of the first, which is why a Q4 model is usually the right starting point on consumer hardware.