llamaperf
← All articles

Reading benchmarks

Speculative decoding: when a draft model speeds you up

By llamaperf · · 5 min read

Quick answer

Speculative decoding has a small, fast model guess the next few tokens, and the big model checks them all in one pass. When the guesses are right you get text sooner, and it's the same text the big model would have written anyway. One user and predictable text get the biggest gain. Whenever you share a speed figure, say whether it was on, because results with and without it can't be compared.

How speculative decoding works

Most of each step a big model takes goes on reading its weights from memory, and checking several tokens in one step costs about the same as writing one. A fast draft model can use that spare capacity to propose tokens.

A small model guesses, the big one checks

The draft model proposes a few tokens. The main model scores all of them at once, keeps the ones it agrees with and writes its own token where the draft went wrong. Several accepted guesses can produce tokens for roughly the cost of one step. Rejected guesses add draft work without speeding up the answer.

vLLM describes the sampling behavior as follows:

Speculative decoding sampling is theoretically lossless up to the precision limits of hardware numerics.

From the vLLM speculative decoding docs

MTP does the same thing from inside the model

Some models ship with extra prediction heads trained to guess the next few tokens themselves, usually called multi-token prediction or MTP. The principle is the same and there's no second model file. When you report a result, MTP counts as speculative decoding too.

When a draft model pays off

Predictable text gains the most

How much you gain depends on how often the draft is right. Code and structured output hand the draft lots of easy wins, as does anything that repeats your input. Creative writing at a high temperature gives it fewer. So two people with the same card and the same pair of models can report very different speedups simply because they sent different prompts.

A busy server gains less

The vLLM docs pitch speculative decoding as a way to

reduce inter-token latency under medium-to-low QPS (queries per second), memory-bound workloads.

From the vLLM speculative decoding docs

One person on a home setup is exactly that case. Once many requests share the GPU, the spare compute that made guessing cheap is already in use and the benefit shrinks. If other people use your model too, test speculation again under that load before you leave it switched on.

Turning it on in llama.cpp

llama-server takes a draft model with -md (also spelled --spec-draft-model), as listed in the server README. The draft has to share the main model's tokenizer, which in practice means a smaller model from the same family.

llama-server -m main-model.gguf -md small-draft.gguf --spec-draft-n-max 3

How many tokens to draft

Per the same README, --spec-draft-n-max defaults to 3. A longer draft wins more when its guesses land and wastes more when they miss, so try a couple of values on your own prompts and keep whichever finishes fastest end to end. llama.cpp renames flags between releases, so run llama-server --help on your build before copying any command.

Check the gain yourself

Run one prompt twice, draft on and draft off, with the same output limit, and compare how long each takes to a finished answer. If the gain is small, try a smaller draft model. A slow draft can cancel the time saved by accepted guesses. Our calculator estimates plain decoding speed only, so a speculative run that beats its figure doesn't mean the estimate is broken.

Say whether it was on when you share a number

With speculation on, a tokens-per-second figure measures something different from a plain run, and mixing the two makes a card look faster or slower than it is.

Why the two numbers can't share a median

Take two made-up figures on the same card and model: one person reports 40 tokens per second with plain decoding and another reports 70 with a draft model. Average them and you've described a setup nobody ran. Anyone comparing cards on the leaderboard or in a head-to-head comparison needs to know which kind of number they're reading.

What to include in your report

Say whether speculation was on and which method you used, and for a draft model, give its exact file and the draft length. Include a plain-decoding run on the same prompt so readers can compare the time with and without speculation. Our benchmark-reading guide lists the other fields worth recording, and the submission form has room for all of it, draft file and length included.

Frequently asked questions

How does speculative decoding work?

A small draft model guesses a few tokens ahead and the main model checks them all in one pass. Every correct guess is kept, which is how you get several tokens for roughly the cost of one step of the big model.

Does speculative decoding lower output quality?

No. The main model still decides every token. vLLM's documentation calls the sampling theoretically lossless up to the precision limits of the hardware.

Which draft model should I use?

A small one from the same family, so the tokenizer matches. Start with the smallest you can find, because a draft that runs slowly cancels out the time it saves.

Is speculative decoding worth it when offloading to CPU?

Maybe, and it's worth testing. With part of the model in system RAM, every step of the main model is slower, so each good guess saves more time. A slow draft can eat that gain, so compare runs with it on and off.

What is the difference between MTP and a draft model?

They do the same job: guess tokens ahead for the main model to check. A draft model is a separate, smaller file. MTP is extra prediction heads trained into the main model itself, so there's no second model to load.

Why do speculative decoding speedups vary so much between reports?

Because the gain depends on how often the draft guesses right, and that shifts with the prompt, the sampling settings and the draft model. Code and other predictable text gain more than creative writing does.