Local model performance: what makes an LLM fast or slow

Generation speed (tokens/s)

How quickly output tokens are produced. Check whether the figure describes one request or totals across concurrent requests.

Prompt processing (tokens/s)

How quickly input tokens are processed. Long documents and repeated agent prompts can spend much of their time here. Cached input can change the result.

Time to first token and completion time

The wait for the first visible output and the wait for the whole answer. Loading, queueing and input processing can contribute; generation speed alone does not measure either.

Memory use and answer quality

Check weights, KV cache, runtime buffers and available memory. Test usefulness on your own tasks: report counts and speed are not accuracy scores.

Check the conditions before comparing two reports →

Start with the model size, quantization and context you need. For a discrete GPU, check both VRAM and any system RAM used for offloading. On a Mac, the OS and other applications share unified memory with the model.

For bandwidth-limited decode, a useful upper-bound intuition is memory bandwidth ÷ bytes moved per output token. Real inference also involves compute, attention, cache access and transfers. This is not a universal speed formula: prompt processing, batching, long context and speculative decoding can behave differently.

• Quantization: fewer bits can reduce weight memory, but speed depends on the runtime and kernels. Halving weight size does not guarantee twice the speed. Compare quant formats.
• Mixture of experts: active parameters affect per-token work, while total parameters still matter for storing the weights. Routing and offloading add costs.
• Context: use the context you actually need when checking fit. A model's advertised maximum is not a promise that your machine can run it at that length.
• Engine: check support for your GPU and quant format. Record the version, backend and settings when comparing results.

The calculator separates estimates from community measurements. Open its breakdown and the underlying setup before treating a speed as a prediction for your workload.

Model-specific VRAM requirements · Mac performance reports · Models people use for coding

These are separate community setups, not controlled hardware comparisons. Rows are ordered by generation report count, then PP report count. Each median pools different model sizes, quants and engines, so it cannot tell you which GPU is faster for the same workload.

Generation and prompt-processing medians are calculated separately from positive measurements in plain, single-device reports. A device needs at least 3 generation reports or at least that many PP reports to appear; each median uses only reports that include its metric. A recorded single GPU does not establish that the model stayed entirely in VRAM or that only one request ran at a time.

See individual measured examples with source links →

This table is generated from the current dataset on each request. Report-added dates describe collection, not necessarily when a benchmark was run. Link to this table; cite the access date and retain the workload caveat.

1. Fix the model file, quant, engine version, hardware and offload settings. Save the prompt and the intended output length.
2. Separate a cold start from warmed-up runs. Keep concurrency explicit and repeat the same workload without changing settings.
3. Record actual input/output token counts, generation and prompt timings, and wall-clock completion time. A configured context limit is a separate field.
4. Keep the raw output and report variation across runs. Leave unmeasured latency or missing settings unknown.

Follow the measurement workflow and copy the report template →

Submit your report with the source, full setup and measurement conditions.

Long wait before output starts

Separate model loading from input processing. Check prompt length, cached input and competing requests before looking only at generation speed.

Slow generation or sudden slowdown

Inspect GPU residency, system-memory use, actual context, power/thermal limits and engine backend. Compare a shorter context with otherwise identical settings before buying hardware.

Another report looks much faster

Match model size, quant, GPU count, concurrency, offload and speculative decoding. If conditions are missing, treat the comparison as incomplete.

It fits but gives poor answers

Test representative tasks and review the model, prompt, quant and runtime settings. Memory fit and token rate do not establish answer quality.

What does local model performance mean?

Here it means the speed and memory requirements of running a model on your own hardware. Generation speed, prompt processing and time to first token describe different parts of the experience. Answer quality needs separate evaluation on your tasks.

What is a good tokens-per-second speed for a local LLM?

It depends on your task and tolerance for waiting. Measure time to first token and total completion time as well as generation speed. Compare runs with the same model, quantization, input length and concurrency; a high aggregate throughput does not describe one conversation.

Does a faster GPU always run local models faster?

No. Memory capacity, bandwidth, compute, engine support and workload all matter. Decode for one request is often bandwidth-limited; prompt processing and concurrent serving can depend more on compute. Offloading and device transfers can change the bottleneck.

Why does a local model slow down with a long context?

Long inputs take time to process and can increase attention work and KV-cache memory during generation. The effect depends on the architecture and runtime. Check actual input tokens separately from the configured context limit.

Does the inference engine change local model performance?

Yes. Kernel support, quant formats, caching, batching and speculative decoding can all change speed and memory use. Compare the same workload and record the engine version and settings before drawing conclusions.

Is a model that fits in memory guaranteed to work well?

No. Fit is a capacity estimate. Runtime allocations and other applications affect available memory, and a model can fit while remaining too slow or inaccurate for your task. Use the calculator to plan, then check measured reports and test your own workload.